WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=uIiA6DquRiE

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/uIiA6DquRiE.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:13.040 --> 00:00:14.320
Hello everyone.

s2
00:00:14.320 --> 00:00:16.800
Um yeah, thanks so much for coming today.

s3
00:00:16.800 --> 00:00:17.840
Much appreciated.

s4
00:00:17.840 --> 00:00:19.359
Um yes, I'm Daniel.

s5
00:00:19.359 --> 00:00:20.720
I'm from Anslaf.

s6
00:00:20.720 --> 00:00:22.400
My brother is also here today.

s7
00:00:22.400 --> 00:00:24.560
Um but yeah, like you know, thanks for coming.

s8
00:00:24.560 --> 00:00:33.600
Um, so for you folks who don't know us, um, so we actually, you know, we're one of the largest distributors of language models and diffusion models as well.

s9
00:00:33.600 --> 00:00:35.040
So we don't just do language models.

s10
00:00:35.040 --> 00:00:37.040
We upload our models to hugging face.

s11
00:00:37.040 --> 00:00:42.800
Um, and you know, we're on the I think we're number 10 or something on the Oh, no, I don't I don't remember.

s12
00:00:42.800 --> 00:00:46.079
But anyways, we're on the list of the top organizations on Hugging Face.

s13
00:00:46.079 --> 00:00:48.399
Um, we have over 300 million total downloads.

s14
00:00:48.399 --> 00:00:50.320
Um, so definitely check us out on that.

s15
00:00:50.320 --> 00:00:56.960
Um you can run like you know Deepseek, GLM, many other models and we quantize them down using dynamic quantization.

s16
00:00:56.960 --> 00:01:00.399
Um so you can run them on your local computer.

s17
00:01:00.399 --> 00:01:03.359
Um we also do many bug fixes for open source models.

s18
00:01:03.359 --> 00:01:14.159
Um so you know we you know fix many bugs in you know OpenAI's GPUs um you know Meta's models um Google's models deepseas many other models we fix bugs in them.

s19
00:01:14.159 --> 00:01:18.640
Um and so like you know they have many issues sometimes and then we post about them on Twitter.

s20
00:01:18.640 --> 00:01:31.360
um you know we post about our findings um so you know m most of the open source models that you probably guys have used um are most likely fixed by us um and yeah like we collaborate with everyone in the entire world

s21
00:01:31.360 --> 00:01:45.360
um on you know model releases um yeah we also collaborate with hardware providers and you know we really appreciate the collaborations with everyone we also don't just do model fixes and bug you know bugs we also introduce new features

s22
00:01:45.360 --> 00:01:48.640
and we also like you know do fixes for the entire trading stack.

s23
00:01:48.640 --> 00:01:54.079
Um for example, we introduced something called async gradient checkpointing which is used by many organizations.

s24
00:01:54.079 --> 00:01:57.759
Um we also introduced flex attention which is used by many folks.

s25
00:01:57.759 --> 00:02:05.040
Um and we also fixed a gradient accumulation bug fix um which increased accuracy by 1 to 3% um across the entire training stack.

s26
00:02:05.040 --> 00:02:14.400
Um so we don't just like you know do bug fixes for models um it's also like you know whole training stack um fixes and stuff like that.

s27
00:02:14.720 --> 00:02:17.440
So today, you know, the workshop is quite long.

s28
00:02:17.440 --> 00:02:20.080
Um, so there will be multiple sections in the workshop.

s29
00:02:20.080 --> 00:02:23.120
Um, and so after each section, anyone can ask a question.

s30
00:02:23.120 --> 00:02:31.040
Um, and so, you know, please, I guess, if I'm not sure if there's a microphone, but if you could raise your voice and you ask a question, you know, I'm more than happy to answer them.

s31
00:02:31.040 --> 00:02:35.680
Um, but you know, the first section we're going to be talking about is the state of AI.

s32
00:02:35.680 --> 00:02:40.239
So, where is currently language models, AI models, where are they at currently?

s33
00:02:40.239 --> 00:02:44.480
Um so I'm not sure if everyone knows the meter plot.

s34
00:02:44.480 --> 00:02:48.400
Um so this meter plot shows the time horizon of uh models.

s35
00:02:48.400 --> 00:02:56.319
Um if you can you know every single task if it takes a human 16 hours can a model you know finish that task.

s36
00:02:56.319 --> 00:03:02.319
Um and you can see on this plot you know cloud mythos you know preview is very good.

s37
00:03:02.319 --> 00:03:06.400
It can do tasks that humans can do that take you know human 16 hours.

s38
00:03:06.400 --> 00:03:08.560
Um you know opus 4.6 is also there.

s39
00:03:08.560 --> 00:03:18.400
you know all the other models are also there and so you know this plot is very good because it symbolizes the AI models are getting better and better and better over time

s40
00:03:18.800 --> 00:03:34.560
you know recently with the launch of you know GPD 5.6 um you know just well their preview model um you know just on Friday um you know I put the plot so they didn't so me didn't actually update their plot um because they said that the results were not trustworthy enough

s41
00:03:34.560 --> 00:03:43.840
um but you know I just put it on the plot um and so you can see that GBD 5.6 you know is around you know opus 4.6 sixth level I guess with large

s42
00:03:43.840 --> 00:03:57.120
confidence bounds um so it's very you know uncertain about the capabilities of the model um however if you include cheating so if you include that the model sometimes likes to cheat on some of the tasks then it actually goes to 270

s43
00:03:57.120 --> 00:04:07.840
hours um so we're directly and you know if you look at the y-axis I actually did a disjoint graph um so the y-axis is 50 hours skipped to 250

s44
00:04:07.840 --> 00:04:15.519
hours um so if you can imagine the graph is actually very skewed Um when I like made the graph, um GBD 5.6 was like a very big outlier.

s45
00:04:15.519 --> 00:04:17.359
Um so I had to like compress the graph.

s46
00:04:17.359 --> 00:04:24.400
Um but this only you know this graph only works if you consider that GBD 5.6 cheated on some of the tasks.

s47
00:04:24.400 --> 00:04:29.840
Um and so we'll be talking about you know why AI models cheat and how do we like you know solve these issues.

s48
00:04:29.840 --> 00:04:35.440
Um but yeah this plot is very useful to showcase the capabilities of these models.

s49
00:04:35.600 --> 00:04:37.680
So previously this is 50%.

s50
00:04:37.680 --> 00:04:49.919
you know if you could if a model can complete the task with 50% of the ch you know of the time to 50% accuracy if you want to actually oneshot the model so you just ask the model you know implement X or implement Y

s51
00:04:49.919 --> 00:04:58.240
um and you want the model to do very well then you want to look at the 80% success rate if you look at the 80% success rate it kind of drops

s52
00:04:58.240 --> 00:05:15.039
quite a lot um so you can see that previously mythos is around 16 17 hours um now it only can do three hours so if you prompt a model and you want to have like a oneshot example, you know, you just trust the model by just or asking it, you know, implement,

s53
00:05:15.039 --> 00:05:20.400
I don't know, page rank or something, you know, implement some sort of rag system, you know, fine-tune a model or something like that.

s54
00:05:20.400 --> 00:05:23.600
Um, it can only do a task that will take a human three hours to do.

s55
00:05:23.600 --> 00:05:26.479
Um, and so the so that is a problem with AI models.

s56
00:05:26.479 --> 00:05:32.479
Um, generally speaking, if you want to use AI models very well, you need to prompt it at least like, you know, five times or something.

s57
00:05:32.479 --> 00:05:46.479
Um and each of those times assuming they're independent um the success rate is much higher if you're prompted many many times right you can't just call the model once and expect it to do work to do well um you need to call it multiple times

s58
00:05:46.479 --> 00:06:00.639
um and you can also work out the probability of it like succeeding you know if the model is 50% accurate um then it will be 50% failure then it's 1 minus 0.5 to the power of five or something like that you know if you do five turns and then your success rate jumps to like 97%

s59
00:06:00.639 --> 00:06:07.039
or something um so you need call the model at least five times for it to be very effective.

s60
00:06:08.080 --> 00:06:20.400
So previously these are linear you know this is a linear trend you know on the y-axis it's just it's not you know it's just linear um if we log it you know if we log

s61
00:06:20.400 --> 00:06:30.639
the y-axis you can see that it's more exponential progress um so it's actually a straight line fit to the entire progress of AI models on the meter time horizon

s62
00:06:30.639 --> 00:06:45.039
um you know benchmark you can see that you know it's very clear that AI models are getting better and better over time um I als We also added you know GBD 5.6 six with the cheating and no cheating and also claude mythos are you know accentuated that and you can see I you don't need now you don't

s63
00:06:45.039 --> 00:06:53.520
need to like you know fake the y-axis you know you don't need to do like a disjoint y-axis um if you do that you can see that you know models are getting better over time

s64
00:06:53.520 --> 00:07:06.880
um and supposedly you know if this trend continues these models will get better and better and better better and much better um yeah so so the question is if the trend continues you know that's the fundamental question

s65
00:07:06.880 --> 00:07:18.800
Um and it's not just you know one specific task for this benchmark that you can see that models are getting better over time across all benchmarks models are getting better over time right so like you know GPQA

s66
00:07:18.800 --> 00:07:30.240
diamond you know it's kind of plateau you know it's kind of already saturated as a benchmark but over time you know it does very well you know every single benchmark you see models are getting better right live code bench you know maths algorith

s67
00:07:30.240 --> 00:07:37.919
maths tests um you know even Tesla's you know you know self-driving I guess is also has like a doubling time of 17 months.

s68
00:07:37.919 --> 00:07:41.360
Um so every single 17 months the models will get better and better.

s69
00:07:41.360 --> 00:07:43.680
Um you know double double their capabilities.

s70
00:07:43.680 --> 00:07:51.919
Um so over time all these models in every single subject you know every single area it will get better.

s71
00:07:51.919 --> 00:08:03.599
Um so I guess the main question is you know if we assume every single subject every single area the models get 100% like you know approaching 100% accuracy is this AGI?

s72
00:08:03.599 --> 00:08:13.280
Um so that is one of the fundamental questions that people ask you know if we just get better on benchmarks um is this AGI um what happens if we get better on all benchmarks

s73
00:08:13.280 --> 00:08:18.080
you know every single benchmark that human humanity has created it just gets better on all of them.

s74
00:08:18.080 --> 00:08:28.479
Um yeah but so this is a you know very good plot show well I guess chart showing all of the different types of benchmarks and they all get better over time.

s75
00:08:28.960 --> 00:08:40.640
everyone's favorite I guess artificial intell uh you know artificial analysis benchmark showing you know artificial intelligence getting much much better over time as well you know fable I guess is I guess the best for now um

s76
00:08:40.640 --> 00:08:52.240
although not everyone can access it currently but anyways it's for now it's the best um and you can see over time that you know these models are getting better over time as well um and you know like this plot showcases

s77
00:08:52.240 --> 00:09:01.440
um a very useful indication you know like how do we like you know benchmark you know is this benchmark actually good um in terms of like you know showcasing the capabilities of models as well.

s78
00:09:01.440 --> 00:09:04.320
Um and we'll be also discussing about that as well.

s79
00:09:04.320 --> 00:09:09.120
Um on the other hand yes models are getting better over time.

s80
00:09:09.120 --> 00:09:15.040
Um but there are some things which models are not very good at still for example long context is not doing very well.

s81
00:09:15.040 --> 00:09:18.800
Um so you know most models you might say okay Gemini has 1 million context length.

s82
00:09:18.800 --> 00:09:20.480
You know GBD has 1 million context length.

s83
00:09:20.480 --> 00:09:22.399
Claude has 1 million context length.

s84
00:09:22.399 --> 00:09:25.680
But should you actually use all of the 1 million context length?

s85
00:09:25.680 --> 00:09:36.320
Um so there are actually benchmarks to showcase that if you use for example GBD 5.5 um you know if you use 512 context your accuracy reduces to 50%.

s86
00:09:36.320 --> 00:09:43.040
Um so if you use you know 512 context you will only remember 50% of the facts that you wrote in the previous context.

s87
00:09:43.040 --> 00:09:46.720
Um so maybe that's not a good idea to use the full context.

s88
00:09:46.720 --> 00:09:53.200
Um you can see opus 4.7 um 4.6 4.7 is the very last orange line.

s89
00:09:53.200 --> 00:09:57.360
Um so at the context length of 256K it goes to 0%.

s90
00:09:57.360 --> 00:09:59.760
Um so this might be a benchmark flaw.

s91
00:09:59.760 --> 00:10:02.640
Um so maybe don't trust the benchmark too much.

s92
00:10:02.640 --> 00:10:05.040
Um but it's good to look at the benchmark overall.

s93
00:10:05.040 --> 00:10:07.760
You know where is the model's capabilities for long context.

s94
00:10:07.760 --> 00:10:11.440
Um the blue lines I highlighted are open source models.

s95
00:10:11.440 --> 00:10:14.640
You know deepseek gl 5.1 other models.

s96
00:10:14.640 --> 00:10:16.320
Green is Google's models.

s97
00:10:16.320 --> 00:10:22.240
Um but you can see in general you know models are models definitely do degrade over long context.

s98
00:10:22.240 --> 00:10:39.040
Um so if you you know for example if you set like a you know automatic compaction area I would not suggest you to use all 1 million context maybe maximum 600k or something um and then compact it and then continue your you know coding session um but I would yeah but in general

s99
00:10:39.040 --> 00:10:50.399
you know this plot shows that long context still has a very long way to go um and if we want to have long context you know capabilities um labs I guess will have a lot of

s100
00:10:50.399 --> 00:10:53.839
time to fix this problem.

s101
00:10:53.839 --> 00:10:54.160
Yeah.

s102
00:10:54.160 --> 00:10:58.000
So another plot is you know just showing open source versus closed source.

s103
00:10:58.000 --> 00:11:02.320
So open source still has some way to go for this you know long context.

s104
00:11:02.320 --> 00:11:06.800
Um so open source is blue line and the black lines are like you know closed source models.

s105
00:11:06.800 --> 00:11:12.560
Um and you can see in general open source does okay but there's definitely much more room for improvement.

s106
00:11:12.560 --> 00:11:15.680
Um I guess compared to Opus 4.7 it's better.

s107
00:11:15.680 --> 00:11:19.680
Um but you know maybe this benchmark does need maybe there are some flaws in the benchmark as well.

s108
00:11:19.680 --> 00:11:27.360
Um yeah, but overall you know this plot shows that long context definitely still has more room for improvement.

s109
00:11:28.480 --> 00:11:39.680
And also you know like if you looked at the plot previously you know this meter plot um I'm not sure if you can see that before 01 preview there is actually a plateau of performance.

s110
00:11:39.680 --> 00:11:45.279
Um and so if you can see you know GBD4 to GBD40 there's not that much performance improvement.

s111
00:11:45.279 --> 00:12:01.040
Um and so this time frame around one year um was when you know the labs were confused on what is next um you know before 01 preview which showed that reasoning was very important they didn't actually know what to pursue next um and so for one year

s112
00:12:01.040 --> 00:12:14.079
the models kind of plateaued um and so I call this the intelligence plateau the hypothesis that you know you know assume that we never have discovered reasoning then maybe air models would have like plateaued

s113
00:12:14.079 --> 00:12:24.240
um but because we have discovered reasoning you know we have shown that models can do reasoning capabilities we have continued the trend continuously um and so normally I don't know if this is like luck

s114
00:12:24.240 --> 00:12:40.240
um or if this is a self fulfilling prophecy um so I don't know if you guys you know the moor law um you know mos law has continued um not because of the law but because people know that it must continue and so people invest money into the resources to make the law continue

s115
00:12:40.240 --> 00:12:47.200
um and so this kind of like shows that you know we might have been in of the world where models have stopped improving.

s116
00:12:47.200 --> 00:12:53.920
Um but you know with the launch of 01 preview you know I guess models have went back to trend.

s117
00:12:53.920 --> 00:13:01.120
In fact I made a plot showcasing you know assuming we did not discover reasoning or 01 preview.

s118
00:13:01.120 --> 00:13:06.399
Um then the black line was the supposed you know capabilities of the models.

s119
00:13:06.399 --> 00:13:10.800
You can see I made it into a S shape um like a you know a sigmoid type shape.

s120
00:13:10.800 --> 00:13:19.760
Um and if you know if we didn't discover reasoning then models definitely will taper off in terms of capabilities right we'll only have a model that's as capable as

s121
00:13:19.760 --> 00:13:30.320
claw 3.7 sonnet I guess or 01 or something like that um but you know luckily because of reasoning and this new paradigm of scaling you know the green line is the new scaling law

s122
00:13:30.320 --> 00:13:41.440
um and you can see previously the black line the doubling time was actually around seven months so every single seven months the capabilities of the models double um but now it has shrunk to 3.5

s123
00:13:41.440 --> 00:13:41.920
months.

s124
00:13:41.920 --> 00:13:47.440
So every single 3.5 months you just need to wait 3.5 months and the models will get double better, right?

s125
00:13:47.440 --> 00:13:48.720
Better by two times.

s126
00:13:48.720 --> 00:13:51.440
Um and that's quite striking I guess.

s127
00:13:51.440 --> 00:13:56.320
Um so the main question though is will the green line continue as a straight line?

s128
00:13:56.320 --> 00:13:59.279
Um that is a fundamental question that labs are still struggling on.

s129
00:13:59.279 --> 00:14:15.680
you know what happens if the green line again you know the green line again goes as a S shape you know that's possible um but you know we don't actually know if this will happen you know if the green line will continue scaling you know going all the way up to infinity I guess or would it be

s130
00:14:15.680 --> 00:14:24.240
like an S shape um and this is you know many researchers are you know I guess have sleepless nights you know what is the next you know what is the next thing afterwards

s131
00:14:24.240 --> 00:14:32.720
after reasoning after 01 you know what is the next thing afterwards um and you know Many researchers will need to like you know I guess think about this.

s132
00:14:32.720 --> 00:14:43.279
Um yeah but you know this plot is very you know this is one of my favorite plots because it shows that you know AI progress can continue over time with new ideas and innovation.

s133
00:14:43.839 --> 00:14:44.480
Oh yes.

s134
00:14:44.480 --> 00:14:47.120
So does anyone have any questions for the first section?

s135
00:14:47.120 --> 00:14:48.560
Um yes.

s136
00:14:48.560 --> 00:14:52.000
So we came all the way to one trillion right?

s137
00:14:52.000 --> 00:15:00.399
Do you think the next jump if we need do we need like 10 trillion parameters when we'll see the jump or hardware will be the limitation that

s138
00:15:00.399 --> 00:15:01.519
yes that's a great question.

s139
00:15:01.519 --> 00:15:09.680
So the question was you know models we're currently at one trillion parameters do we need to go to 10 trillion parameters or more for models to be even more capable?

s140
00:15:09.680 --> 00:15:20.639
Um so the scaling laws does say that you know if you multiply the parameters and the data size um generally speaking this number if you increase the number you will get the models become more capable.

s141
00:15:20.639 --> 00:15:26.240
So yes you can increase the parameters by 10 times and in general your performance will increase.

s142
00:15:26.240 --> 00:15:30.800
Um however the view is there is going to be diminishing returns.

s143
00:15:30.800 --> 00:15:35.680
Um I feel like you know it's not just the model size times the data set size.

s144
00:15:35.680 --> 00:15:39.519
It's actually a ratio um some sort of like power law when you multiply them.

s145
00:15:39.519 --> 00:15:41.519
So you actually get diminishing returns over time.

s146
00:15:41.519 --> 00:15:42.399
So yes, you're right.

s147
00:15:42.399 --> 00:15:48.800
If you want to have actually I'm not sure the exact law, but if you want to have double capabilities, you do need to 10 times the parameters.

s148
00:15:48.800 --> 00:15:52.240
Um and then if you want another double, you have to 10 times it again.

s149
00:15:52.240 --> 00:15:55.920
So it's 1 to 10 to 100 trillion parameters.

s150
00:15:55.920 --> 00:15:59.839
um if you want maybe that's not a good way to scale.

s151
00:15:59.839 --> 00:16:08.959
Um maybe instead you know instead of making a 100 trillion parameters some sort of new algorithm or new architecture could solve that problem.

s152
00:16:08.959 --> 00:16:17.120
Um but you're right like if you're a lab you want to do something easy and so the easiest path is to just make a 10 trillion parameters.

s153
00:16:17.120 --> 00:16:22.639
Um but I would say like you know maybe a new algorithm will be better.

s154
00:16:22.639 --> 00:16:27.120
Um yeah any other questions?

s155
00:16:27.120 --> 00:16:27.839
Yes.

s156
00:16:27.839 --> 00:16:33.600
So you do think that we are approaching the limitation of next token prediction.

s157
00:16:33.600 --> 00:16:34.800
That is a good question.

s158
00:16:34.800 --> 00:16:44.800
I would say that for next token prediction it's very powerful because you can essentially the human language is extremely powerful and it doesn't have to be human language.

s159
00:16:44.800 --> 00:16:59.360
It can be you know maths coding you can just predict the next word and in order to predict the next word or token you need to know everything about that token or that word right so like I think IA was talking about like you know Ilia Satska he was saying like you know you need to have you

s160
00:16:59.360 --> 00:17:11.760
need to make a weld model in the model in order to like predict the next word so I still think next word prediction still has a lot of way to go for example if you see this plot you know if we didn't have reasoning

s161
00:17:11.760 --> 00:17:21.520
I guess okay maybe it would have plateaued But because we have discovered this new methodology you know reasoning and trying to like scale even more on next word you know next word prediction

s162
00:17:21.520 --> 00:17:31.280
we have you know went back to trend um I feel like so the main question is if we don't have next word prediction what is next um that is the fundamental question most

s163
00:17:31.919 --> 00:17:40.640
I mean I'm not sure like you know I'm not certain what's what's the next thing I feel like next word prediction is just extremely powerful because it's very easy to formulate

s164
00:17:40.640 --> 00:17:45.679
and you can just like you know you can have like you know because attention is very powerful as well.

s165
00:17:45.679 --> 00:17:49.919
You can have, you know, this special cause of attention mechanism and it's very efficient to train.

s166
00:17:49.919 --> 00:17:53.039
So, I'm not sure I think the main question is I'm not sure what's next.

s167
00:17:53.039 --> 00:17:58.640
Um, I guess researchers will like, you know, they're trying to scratch their heads, you know, what is next afterwards?

s168
00:17:58.640 --> 00:18:00.559
Um, yeah.

s169
00:18:00.559 --> 00:18:01.600
I I Yeah.

s170
00:18:01.600 --> 00:18:02.320
Yes.

s171
00:18:02.320 --> 00:18:03.360
Just a follow up on it.

s172
00:18:03.360 --> 00:18:09.440
Do you feel like we are in the same era like how we attention came out?

s173
00:18:09.440 --> 00:18:09.760
Right.

s174
00:18:09.760 --> 00:18:14.000
So, attention like we don't know what's next.

s175
00:18:16.559 --> 00:18:18.000
Yes, that's a fair followup.

s176
00:18:18.000 --> 00:18:21.520
So, um, you were mentioning how it's kind of like LSTMs or the old AI world.

s177
00:18:21.520 --> 00:18:23.120
We don't know what's next afterwards.

s178
00:18:23.120 --> 00:18:24.799
Um, that's a fair point.

s179
00:18:24.799 --> 00:18:30.559
I feel like so like, you know, previously this example, right?

s180
00:18:30.559 --> 00:18:40.480
So, after GBD4, it was just pre-training, some, you know, supervised fine tuning, some ROF, you know, some RL um, and they waited one year until 01 preview.

s181
00:18:40.480 --> 00:18:48.480
So in this one year of fog you know the fog of war we don't know what what was next and so researchers you know were scrambling you know do we do the reasoning

s182
00:18:48.480 --> 00:19:04.559
process do we make pre-training better do we make the model bigger and bigger and bigger you know they tried all these experiments um and reasoning was the one that won I guess um but I think like I think the main question is is the green trend going to continue

s183
00:19:04.559 --> 00:19:16.640
at the current time it looks like it's continuing once we see models starting to taper out in intelligence, you know, in capabilities, then we'll go back to the, you know, olden days of like, you know, this one year waiting period.

s184
00:19:16.640 --> 00:19:19.200
But I think for now, these models seem very powerful.

s185
00:19:19.200 --> 00:19:21.039
Um, yeah.

s186
00:19:21.039 --> 00:19:26.320
So, like I'm not sure if this will, I mean, if you look, okay, if you squint at the plot, I guess maybe we're tapering out.

s187
00:19:26.320 --> 00:19:30.559
Maybe um let's not consider the GBD 5.6 cheating example, right?

s188
00:19:30.559 --> 00:19:31.760
Let's remove that from the plot.

s189
00:19:31.760 --> 00:19:39.600
Um, but you can see the GBD 5.6 Mythos, you know, 4.6. They're kind of all I guess they're kind of tapering.

s190
00:19:39.600 --> 00:19:46.559
Um so maybe as a I mean I don't know if we someone wants to bet on this but you know maybe models have tapered out but we're not sure.

s191
00:19:46.559 --> 00:19:48.480
So we shall wait a few more months and see.

s192
00:19:48.480 --> 00:19:50.640
So let's wait 3.5 months.

s193
00:19:50.640 --> 00:19:55.120
If we wait 3.5 months and see the models do not improve then we have tapered out.

s194
00:19:55.120 --> 00:19:57.520
Um but remember we only need to wait 3.5 months.

s195
00:19:57.520 --> 00:19:59.200
Um so then this law will fail.

s196
00:19:59.200 --> 00:20:10.240
In fact, if you wait seven months, if you wait seven months, so double the time and models have, you know, just assume you know that dotted line that if if the models just follow the dotted line, okay, then we have tape it out.

s197
00:20:10.240 --> 00:20:15.919
And I would agree that, you know, we'll have to design something new in, you know, make some new invention or something like that.

s198
00:20:15.919 --> 00:20:19.200
Um, but for now, you know, for now looks like it's doing fine.

s199
00:20:19.200 --> 00:20:20.799
Um, yeah.

s200
00:20:20.799 --> 00:20:22.160
Okay, next section.

s201
00:20:22.160 --> 00:20:26.240
Um, so every single section we can have questions, so you can ask as many questions as you like.

s202
00:20:26.240 --> 00:20:30.880
Um the next section we're going to talk about is open versus closed models.

s203
00:20:30.880 --> 00:20:36.080
Um so artificial analysis has this very cool plot showcasing the performance of open source.

s204
00:20:36.080 --> 00:20:37.600
So open source is the blue line.

s205
00:20:37.600 --> 00:20:41.200
Um so open source is the blue line and closed source models is the black line.

s206
00:20:41.200 --> 00:20:45.039
Um and you can see that open source does lag.

s207
00:20:45.039 --> 00:20:47.840
You know open source definitely lags over time.

s208
00:20:47.840 --> 00:20:51.679
Um another very good benchmark is called the weird ML benchmark.

s209
00:20:51.679 --> 00:21:03.120
Um this also shows that open source models lag closed source models right the blue line is open source models the green line is closed source models um and you can see over time you know

s210
00:21:03.120 --> 00:21:17.919
the x-axis is release date of the model and the y-axis is performance and you can see that open source models kind of lag close source models and why the weird ML benchmark I'm not sure if you folks actually know about this why the weird ML benchmark

s211
00:21:17.919 --> 00:21:28.720
um it seems like the weird ML benchmark is a very good indicator better than other benchmarks and the reason the reason why is you know previously I mentioned you know previously this graph right

s212
00:21:28.720 --> 00:21:41.440
reasoning the reasoning models are the green line and in the black models are the non-reasoning models and you can see that reasoning models double you know reduce the time of doubling time to 3.5 months previously it was 7 months

s213
00:21:41.440 --> 00:21:51.440
um interestingly on the weird ML benchmark these reasoning models didn't actually do better it didn't actually change the trend um all it did was make slightly better.

s214
00:21:51.440 --> 00:21:56.000
Um and so this weird ML benchmark seems to be more robust.

s215
00:21:56.000 --> 00:21:58.799
Um and that is why you know this benchmark is very useful.

s216
00:21:58.799 --> 00:22:11.200
Um in fact if you go on the Twitter bus um before GLM 5.2 got released um most you know most of the Twitter people said oh you know deepseek you know deepse if you see very if you squint okay I think I have a plot.

s217
00:22:11.200 --> 00:22:16.320
Oh yes if you squint deepseek and Kimmy are in that little corner over there.

s218
00:22:16.320 --> 00:22:28.640
um you know deepseek those three models the three whales are deepseek you know flash deepseek pro I think one of them's max mode or something like that um and also Kim's over there as well so before GLM 5.2 two got released,

s219
00:22:28.640 --> 00:22:33.760
you know, on the Twitter verse, everyone kept saying that open source models are much worse than closed source models, right?

s220
00:22:33.760 --> 00:22:38.880
They're not lagging, you know, they're not just lagging, they're much worse because of this benchmark.

s221
00:22:38.880 --> 00:22:54.000
Um, in fact, if you look very closely of the weird ML benchmark, all of the top models are closed source labs, you know, like Fable, you know, GPD 5.5, whatever, you know, all of these are just very, you know, it shows very clear that

s222
00:22:54.000 --> 00:22:57.200
open source models are not doing very well in terms of this benchmark.

s223
00:22:57.200 --> 00:23:08.799
um until GPD 5.2 came along um you know number 15 is GPD 5.2 and it shows that actually open source has came back um and GPD 5.2 too kind of shocked the world.

s224
00:23:08.799 --> 00:23:11.760
Um that you know I guess open source has not died.

s225
00:23:11.760 --> 00:23:22.720
Um and you know deep yeah so in general this worked very well you know deepse you know GLF2 showed that you know open source does very well still.

s226
00:23:23.120 --> 00:23:25.360
You can also filter out by country.

s227
00:23:25.360 --> 00:23:30.400
So by country you can see that the black line is United States you know the US models.

s228
00:23:30.400 --> 00:23:34.400
Um the dark red line is the Chinese labs.

s229
00:23:34.400 --> 00:23:36.720
Um and you know there's other labs as well.

s230
00:23:36.720 --> 00:23:39.360
um you know French, South Korean labs and stuff like that.

s231
00:23:39.360 --> 00:23:45.520
Um but you know over time it shows that these models um you know the US labs seem to do very well over time.

s232
00:23:45.520 --> 00:23:51.200
You know they they're always at the frontier and then the Chinese labs like to catch up over time.

s233
00:23:52.320 --> 00:23:58.240
Previously you know I mentioned you know the um you know the plateau before you know before 01 preview got released.

s234
00:23:58.240 --> 00:24:03.440
If you actually look at this plot um there is something called the open source draft.

s235
00:24:03.440 --> 00:24:11.520
Um so after 01 preview got released open source labs did not know how to replicate 01 preview.

s236
00:24:11.520 --> 00:24:14.000
They have never you know they don't know what is reasoning.

s237
00:24:14.000 --> 00:24:21.520
So I'm not sure if you okay this is a few years back um but on Twitter you know open kept talking about oh you know 01 preview was extremely powerful

s238
00:24:21.520 --> 00:24:26.159
um you know every single tweet you see every single day you know they show that 01 preview was very powerful.

s239
00:24:26.159 --> 00:24:35.120
Um and so for one I think it was six months to eight months um open source models open source labs they got confused on what to do next.

s240
00:24:35.120 --> 00:24:48.880
Um but then as everyone knows deepseek R1 came along um and they showed that even for open source models you can train these models to do reasoning gpo um reinforcement learning and it does very very well.

s241
00:24:48.880 --> 00:24:54.960
In fact, if you take this plot, you know, the the black line minus the blue line, if you just minus it, you get this plot.

s242
00:24:54.960 --> 00:24:59.200
Um, and you can see this is how many months behind open source is.

s243
00:24:59.200 --> 00:25:07.679
Um, and you know, over time, um, you can see like, you know, after 01 preview got released, um, you know, it kind of skyrocketed.

s244
00:25:07.679 --> 00:25:13.760
Um, you know, the open source models were very, very lagging in terms of, you know, behind closed source models.

s245
00:25:13.760 --> 00:25:22.159
Um and so like when Deepseek R1 got released then the open source labs knew okay we can also do uh 01 type reasoning.

s246
00:25:22.159 --> 00:25:29.600
Um and that is why recently you know the the time between closed source labs and open source labs have started decreasing again.

s247
00:25:29.600 --> 00:25:33.120
Um yeah so this is slightly outdated.

s248
00:25:33.120 --> 00:25:33.840
This is like May.

s249
00:25:33.840 --> 00:25:38.720
Um so I think now it's actually four months with the release of GLM 5.2. It's around four months now.

s250
00:25:38.720 --> 00:25:45.039
Um so open source labs lag behind closed source labs by around four months.

s251
00:25:45.120 --> 00:25:48.159
There was actually a very nice plot you know doing some sort of regression.

s252
00:25:48.159 --> 00:25:50.720
So some sort of like trend extrapolation.

s253
00:25:50.720 --> 00:26:00.480
Um according to this plot um if you extrapolate the trend by December this year open source models will 100% catch up to closed source models by this year December.

s254
00:26:00.480 --> 00:26:12.159
Um but you know who knows I guess maybe maybe open source maybe we can have an open source model as powerful as the best closed source model by December

s255
00:26:12.159 --> 00:26:23.279
um you know if this trend continues um so I guess the question is will the trend continue um it's always about will the trend continue um and you know may

s256
00:26:23.279 --> 00:26:38.000
most of you maybe may know that you know open source some of the open source improvements in technology you know improvements in capabilities are via distillation you know so some of the open source labs what they like to do is they like to call the models you know core the frontier models like opus

s257
00:26:38.000 --> 00:26:47.679
or GPD and then use the traces to train your model um so this is a common methodology that labs like to do um I wouldn't say this is a bad method

s258
00:26:47.679 --> 00:26:57.679
um but it is a method that you know some closed source labs like to look down upon you know they like to stop you know their view is you know we should not allow these open source labs to like do this training

s259
00:26:57.679 --> 00:27:01.520
um and you know get away for free I guess in terms of training cost.

s260
00:27:01.520 --> 00:27:04.080
Um but you don't actually have to do this approach.

s261
00:27:04.080 --> 00:27:08.559
Um so most labs when you do distillation there are two different types of approaches.

s262
00:27:08.559 --> 00:27:11.520
Um the first approach is you need to have the logits.

s263
00:27:11.520 --> 00:27:14.240
You need to actually have access to the full logs.

s264
00:27:14.240 --> 00:27:18.080
Um and unfortunately most labs do not actually have that right.

s265
00:27:18.080 --> 00:27:20.320
So like labs will not give you the full logs.

s266
00:27:20.320 --> 00:27:26.640
Um instead you only get the reasoning traces that are summarized um and the final output.

s267
00:27:26.640 --> 00:27:32.880
Um, and so these, you know, these open source labs, they're not just, you know, they're not just training on the, you know, Opus output, right?

s268
00:27:32.880 --> 00:27:34.080
That's just, that's silly.

s269
00:27:34.080 --> 00:27:38.880
What they do is they use GRPO or reinforcement learning to recreate the traces.

s270
00:27:38.880 --> 00:27:42.559
Um, and so because you you have the final output, which is the answer.

s271
00:27:42.559 --> 00:27:47.600
All you need to do is use GPU and RL to create the reasoning trace automatically.

s272
00:27:47.600 --> 00:27:50.080
Um, and so that's kind of how they train these models.

s273
00:27:50.080 --> 00:27:55.279
Um, and so you don't actually need to like access the logits or the weights of the model.

s274
00:27:55.279 --> 00:27:56.720
um that's not necessary.

s275
00:27:56.720 --> 00:28:09.120
Um yeah, and you know, one of the most important factors of you know, these large models is, you know, as models get bigger and bigger and bigger, you can't run them on your local device anymore.

s276
00:28:09.120 --> 00:28:11.200
Um it's extremely complicated to run.

s277
00:28:11.200 --> 00:28:17.919
Um and so we do something called dynamic quantization where essentially you take a model, you quantize them down to one bit.

s278
00:28:17.919 --> 00:28:22.320
Um but the trick is you don't quantize every single layer to one bit.

s279
00:28:22.320 --> 00:28:27.440
you quantize some important layers to 16 bit or 8 bit or something like that.

s280
00:28:27.440 --> 00:28:34.240
Um and so if you quantize the whole model down to one bit you will get 0% accuracy right 0%.

s281
00:28:34.240 --> 00:28:42.480
Um but the trick is if you do dynamic quantization so if you look on the you know this is a three-bit Deepseek model um a three-bit one you get 75.6%

s282
00:28:42.480 --> 00:28:45.039
6% accuracy, a three-bit one.

s283
00:28:45.039 --> 00:28:49.200
In fact, if you do dynamic one bit, um you get 57% accuracy.

s284
00:28:49.200 --> 00:28:57.200
Um so we show that you know if you do something called dynamic quantization where you quantize the model down smartly, you can recover accuracy.

s285
00:28:57.200 --> 00:29:03.919
Um and this methodology will become even more important when models get larger and larger and larger and larger.

s286
00:29:04.480 --> 00:29:17.360
if you plot the paro you know efficiency um there if you don't do dynamic if you do you know some other dynamic quantization methods it does okay um but we showed that if you smartly choose the layers it does even better

s287
00:29:17.360 --> 00:29:25.760
um I'm not sure if you folks have followed but GLM 5.2 we also released dynamic quantizations for that we showed that GLM 5.2 2 can quantize very well.

s288
00:29:25.760 --> 00:29:28.799
So if you look I think this is oh this is an animation.

s289
00:29:28.799 --> 00:29:29.679
Oh it works.

s290
00:29:29.679 --> 00:29:38.640
Um but yes you can show the animation you you can see the animation a one bit GLM 5.2 model and this is one bit um and the one bit model

s291
00:29:38.640 --> 00:29:42.159
is literally 86% smaller.

s292
00:29:42.159 --> 00:29:46.399
Um so it's 86% smaller than the full 1.5 terabytes.

s293
00:29:46.399 --> 00:29:50.080
Um and it still managed to do very well on one of the prompts.

s294
00:29:50.080 --> 00:29:52.640
Um so it shows that the models are not dumb.

s295
00:29:52.640 --> 00:29:53.120
Right?

s296
00:29:53.120 --> 00:29:58.000
If you make the model 86% smaller, it does not get 86% dumber.

s297
00:29:58.000 --> 00:30:00.880
Um, it only gets 14% less dumb.

s298
00:30:00.880 --> 00:30:08.080
Um, and so it shows that, you know, if you do special tricks to compress the model, the model still works very well.

s299
00:30:08.080 --> 00:30:20.480
Um, and we also compared to Opus, you know, we compared to Opus 4.8, we compared to GBD 5.5. And also, you have to notice that for G 5.2, I use high reasoning mode.

s300
00:30:20.480 --> 00:30:22.640
You know, for Opus, it's extra high.

s301
00:30:22.640 --> 00:30:25.279
And for you know GPD 5.5 is also extra high.

s302
00:30:25.279 --> 00:30:30.080
Um and so like you know there are different reasoning modes as well which we can also um see.

s303
00:30:30.080 --> 00:30:35.360
Um and all of these are oneshot um so we do not like prompt the model like you know 50 times or something.

s304
00:30:35.360 --> 00:30:39.039
Um this is just one shot directly.

s305
00:30:39.760 --> 00:30:44.399
Okay so the next I guess the open source versus close source section is done.

s306
00:30:44.399 --> 00:30:47.039
I guess any other questions?

s307
00:30:47.039 --> 00:30:47.440
Yes.

s308
00:30:47.440 --> 00:30:53.039
Um so the question was which parts of the model do we quantize to lower bits versus higher precision.

s309
00:30:53.039 --> 00:30:57.360
Um so in general um we did actually a lot of research on this.

s310
00:30:57.360 --> 00:31:03.520
So if you look at the quen the quen 3.5 architecture there are some layers which is the linear attention layers.

s311
00:31:03.520 --> 00:31:07.200
Um the linear attention layers should never be quantized.

s312
00:31:07.200 --> 00:31:11.760
If you quantize the linear attention layers down you will definitely suffer in long context.

s313
00:31:11.760 --> 00:31:18.159
Um so in general the linear attention layers need to be left in 8 bit or 16 bit.

s314
00:31:18.159 --> 00:31:27.200
Um that's for example um another like if you look at the model layers um some layers can be quantized down heavily to one bit.

s315
00:31:27.200 --> 00:31:31.360
Um and the reason why is because these layers are kind of like filler layers.

s316
00:31:31.360 --> 00:31:33.760
Um and so they don't actually do anything.

s317
00:31:33.760 --> 00:31:39.840
Um and in order to check whether a layer does something or not, you do need some sort of collaboration data set.

s318
00:31:39.840 --> 00:31:57.200
So you need you need to have some sort of like representative data and pass it into the model um and you can get you can get the um outputs after each layer and then you can see okay does this model at this specific layer you know does it change that much um and if it doesn't change that

s319
00:31:57.200 --> 00:32:10.240
much okay maybe just quantize a layer to one bed um but if it does change dramatically then you need to be careful um you you cannot quantize that down to like one bed or whatever um so there are actually many we actually publish a lot of like blog

s320
00:32:10.240 --> 00:32:11.279
research on this.

s321
00:32:11.279 --> 00:32:16.960
Um we show I think there was we also show for example you cannot quantize the vision layers down.

s322
00:32:16.960 --> 00:32:21.039
If you quantize the vision layers down you will make the model really bad.

s323
00:32:21.039 --> 00:32:26.559
Um if you give it a you know if you give it a picture of a train it will say it looks like a beach for example.

s324
00:32:26.559 --> 00:32:31.440
Um and so it's you should never quantize the vision layers, the audio layers.

s325
00:32:31.440 --> 00:32:36.320
Um and only the language the language model layers you can like quantize.

s326
00:32:36.320 --> 00:32:38.480
Um but there are many tricks in order to do that.

s327
00:32:38.480 --> 00:32:40.880
Um yeah

s328
00:32:41.039 --> 00:32:41.440
correct.

s329
00:32:41.440 --> 00:32:51.279
So the question was if you do distillation um you you might have done worse on other topics um but you know only if you for example if you just do coding

s330
00:32:51.279 --> 00:32:54.080
it will just do good encoding and then the rest gets very dumb.

s331
00:32:54.080 --> 00:32:55.440
Um so that's a fair point.

s332
00:32:55.440 --> 00:33:00.720
Um so I think that the main trick is you will need to do many many many examples.

s333
00:33:00.720 --> 00:33:03.279
You will call the model like you know 10 million times.

s334
00:33:03.279 --> 00:33:16.159
Um and so like the trick is once you call the middle model 10 million times with high diversity of questions in general um by using the pre-training argument um the model will do well on other tasks.

s335
00:33:16.159 --> 00:33:24.799
Um so the reason why pre-training does very well um is because it has learned so many tasks that it can interpolate the missing holes.

s336
00:33:24.799 --> 00:33:32.480
Um for example, if you just train if you just pre-train a model with just maths questions um assume you do only maths.

s337
00:33:32.480 --> 00:33:34.159
Okay, maybe it's not going to do very well, right?

s338
00:33:34.159 --> 00:33:36.240
And it's not going to do very well on every other task.

s339
00:33:36.240 --> 00:33:44.640
But the trick of pre-training is it does maths, coding, law, you know, every single imag, you know, every single topic you can imagine.

s340
00:33:44.640 --> 00:33:50.000
And the trick is because it has so much knowledge, it feels the holes of the things that it doesn't know.

s341
00:33:50.000 --> 00:33:53.519
Um, and so for distillation, you also need to do the same approach.

s342
00:33:53.519 --> 00:33:56.399
You need to sample, you need to sample well.

s343
00:33:56.399 --> 00:34:03.519
Um so for example instead of doing 10 trillion tokens sample you know like 1% um and then call the model.

s344
00:34:03.519 --> 00:34:06.159
Um yeah so that's kind of how the labs are doing that.

s345
00:34:06.159 --> 00:34:08.240
Um that is a very good question.

s346
00:34:08.240 --> 00:34:13.919
So instead of doing one big quantization can you instead prune the model like you know delete some layers entirely.

s347
00:34:13.919 --> 00:34:18.879
Um so in general from our research pruning does work.

s348
00:34:18.879 --> 00:34:20.639
There is a very big problem though.

s349
00:34:20.639 --> 00:34:22.560
You need to retrain the model.

s350
00:34:22.560 --> 00:34:27.679
you need to continuously train a model after pruning because you have deleted an entire layer.

s351
00:34:27.679 --> 00:34:38.320
Um and so if you delete an entire layer, you will need to do like you know qat or further fine-tuning to push the other to push the other weights to have more knowledge.

s352
00:34:38.320 --> 00:34:48.240
So that is the only problem where if you delete layers um if you don't delete layers when you do you know dynamic dynamic quantization it's called post training quantization.

s353
00:34:48.240 --> 00:34:51.760
So PTQ you do not need to do any training at all.

s354
00:34:51.760 --> 00:35:07.200
if you do you know quantization um but if you do prove the layers you do need to train um so that is one of the problems um yeah yes that's a great question so the question was you know because open source labs you know they use closed source models

s355
00:35:07.200 --> 00:35:18.560
the gap will never actually go to zero um and so I partially agree and so the main argument was labs open source labs the easiest way is to do distillation

s356
00:35:18.560 --> 00:35:30.960
however you know if you for example if you were an open source app you will only use that approach to firstly enter the market but as longterm you know as long-term safety as a long-term safety net you will not do this approach

s357
00:35:30.960 --> 00:35:42.880
um instead as a you know instead you will do you know for example generate the answer get the question for example you know you will get data from a call or scale whatever you know have some sort of like large data labeling

s358
00:35:42.880 --> 00:35:53.280
army or something I don't know um and so like in general because currently some of the labs so they don't just do distillation right so they're not just going to call the model 10 trillion times

s359
00:35:53.280 --> 00:35:54.960
you know and just do distillation.

s360
00:35:54.960 --> 00:35:57.920
They also augment the training data with their own approach.

s361
00:35:57.920 --> 00:36:07.040
So I will be talking about the GLM approach maybe like later um but they did invent some new approaches to do very good reinforcement learning and GRPO um and

s362
00:36:07.040 --> 00:36:14.480
because GRPO and reinforcement learning um you know is open source these labs just use these methodologies to make the models better.

s363
00:36:14.480 --> 00:36:18.560
So you don't so distillation is only one part of the training system.

s364
00:36:18.560 --> 00:36:22.880
Um and it's not I would say that assume distillation disappeared.

s365
00:36:22.880 --> 00:36:36.400
Okay, maybe open source labs maybe increase you know it's not four four months maybe eight months um but but that's fine because you know we always have some sort of innovative and new approach you know deepseek might invent something new

s366
00:36:36.400 --> 00:36:47.359
um and so like you know GLM ki all of them Google you know even the American open source lab they'll have some new innovation um and so like I think like yes if you stop distillation

s367
00:36:47.359 --> 00:36:56.560
it will increase you know the you know four months to eight months but I still think that is fine Um it's just a delay you know and then the delay will go back to like four months.

s368
00:36:56.560 --> 00:37:00.800
Um yeah good question.

s369
00:37:00.800 --> 00:37:05.839
So the question is if dynamic quantization is always better why do people not always do dynamic quantization?

s370
00:37:05.839 --> 00:37:11.280
Um so it depends on the definition of dynamic quantization.

s371
00:37:11.280 --> 00:37:15.599
So for every single lab they will have different approaches to dynamic quantization.

s372
00:37:15.599 --> 00:37:17.119
In fact, I'm actually going to talk about that.

s373
00:37:17.119 --> 00:37:22.745
Um, I was going to talk about that in the benchmaxing and accuracy minimizing session.

s374
00:37:22.745 --> 00:37:24.480
[clears throat] So, I'll be talking about that.

s375
00:37:24.480 --> 00:37:28.400
Um, so I will your question will be answered later.

s376
00:37:28.400 --> 00:37:29.359
Um, yes.

s377
00:37:29.359 --> 00:37:29.680
Okay.

s378
00:37:29.680 --> 00:37:30.320
One more question.

s379
00:37:30.320 --> 00:37:31.680
Yes.

s380
00:37:31.680 --> 00:37:32.000
Yes.

s381
00:37:32.000 --> 00:37:39.680
So, the question was for consumer grade GPUs, you know, what are the open source models in terms of like, you know, the parameter size capabilities and stuff like that.

s382
00:37:39.680 --> 00:37:51.280
Um, so for the open source community, you know, the most popular models are probably Quen 3.6 6 35 billion um 27 billion GMA you know Gemma's 26 billion um GLM

s383
00:37:51.280 --> 00:38:03.440
4.7 flash the smallest type models um and I feel like these small models are actually very powerful um so okay I don't have wait I don't think so I have a plot um but essentially

s384
00:38:03.440 --> 00:38:11.520
these small models the biggest problem oh actually I'm going to talk about this as well the biggest problem of these small models are they fail very bad at tool calling

s385
00:38:11.520 --> 00:38:22.480
because they have tool calling issues um they loop continuously um and the biggest problem is because they're small and that is why they have these problems um but we can counteract this

s386
00:38:22.480 --> 00:38:38.240
um and so one of the things I'm going to talk about later is the model becomes not important anymore it's the harness or the tool that is actually the most important thing um how do you actually call the model um that actually affects the most accuracy of the model um so not actually the model itself

s387
00:38:38.240 --> 00:38:43.599
um but I'll be talking about that as well um yeah okay I will continue on.

s388
00:38:43.599 --> 00:38:45.839
Um, there were always questions after each section.

s389
00:38:45.839 --> 00:38:49.920
Um, uh, yes.

s390
00:38:49.920 --> 00:38:50.400
Oh, yes.

s391
00:38:50.400 --> 00:38:53.599
The next section, the fun section, throughput maxing.

s392
00:38:53.599 --> 00:38:54.560
Oh, actually, I think I did.

s393
00:38:54.560 --> 00:38:55.839
It's supposed to be 2x.

s394
00:38:55.839 --> 00:38:56.400
I don't know.

s395
00:38:56.400 --> 00:38:56.880
Whatever.

s396
00:38:56.880 --> 00:38:59.599
Throughput maxing and and accuracy minimizing.

s397
00:38:59.599 --> 00:39:02.240
I thought it was like accuracy mining, but there's no such thing.

s398
00:39:02.240 --> 00:39:04.720
So, it's called accuracy minimizing for now.

s399
00:39:04.720 --> 00:39:07.599
Um, yes.

s400
00:39:08.400 --> 00:39:11.200
So, this part actually I really like.

s401
00:39:11.200 --> 00:39:11.440
Okay.

s402
00:39:11.440 --> 00:39:14.079
I'm not sure if you guys can see it's a bit oh whatever.

s403
00:39:14.079 --> 00:39:19.040
Um this shows the parade efficiency of cost of the cost of the model.

s404
00:39:19.040 --> 00:39:22.320
So cost is um cost is the x-axis.

s405
00:39:22.320 --> 00:39:25.359
Um and the y-axis is the arena score.

s406
00:39:25.359 --> 00:39:27.680
So this is like you know Ella Marina's arena score.

s407
00:39:27.680 --> 00:39:30.400
Um and this part I really like.

s408
00:39:30.400 --> 00:39:35.760
So I don't really you know maybe you see like arena scores you know Ella Marina scores between each model.

s409
00:39:35.760 --> 00:39:36.560
I don't really like that.

s410
00:39:36.560 --> 00:39:38.640
It's not it's not very easy to see.

s411
00:39:38.640 --> 00:39:46.400
Instead the better approach is to plot every single model on two axes cost versus accuracy.

s412
00:39:46.400 --> 00:39:50.800
Um and you can see Fable does very well right so Fable does very very well on that plot.

s413
00:39:50.800 --> 00:39:58.960
Um but you can see there is a paro trend you know like Gemini 3.1 preview is over here you know Opus 4.6 is over there as well.

s414
00:39:58.960 --> 00:40:04.400
There are some other models as well when Fable got released.

s415
00:40:04.400 --> 00:40:04.720
Okay.

s416
00:40:04.720 --> 00:40:15.200
Well, now it's banned, but anyways, when Fable was released, when you know when people tried it, they noticed that it's not that much better in terms of actual capabilities.

s417
00:40:15.200 --> 00:40:21.440
Um, you can see, you can see, but however, people really liked the front-end design.

s418
00:40:21.440 --> 00:40:28.320
You know, they said if you called Fable, it was very, very good for UI UX front end.

s419
00:40:28.320 --> 00:40:35.040
Um and in fact if you look at the LM Mariners chart you can see it it was a very big shift in terms of front-end design.

s420
00:40:35.040 --> 00:40:40.480
Um GLM 52 is also there if you can see you know it was part of the par paro trend.

s421
00:40:40.480 --> 00:40:49.440
Um but in general for these large models they seem to have they're not going to be do they're not going to be doing that much better on general tasks.

s422
00:40:49.440 --> 00:40:55.119
Um however for UI and designing Fable seems to have done very very well.

s423
00:40:55.119 --> 00:40:57.680
Um, and so you should use Fable for your designing.

s424
00:40:57.680 --> 00:41:08.240
You you should use Fable for designing, for UI, for UX, whatever, HTML, JavaScript, but you should probably not use Fable for the rest of the tasks because it is very expensive.

s425
00:41:08.240 --> 00:41:12.480
Um, so you know, use some other models instead.

s426
00:41:12.480 --> 00:41:23.280
Um, and you know, however, yes, okay, you know, some other models, you know, like, okay, this, you know, this shows that Fable does very well on UI and UX.

s427
00:41:23.280 --> 00:41:35.359
Um but how about over time um you know how what do you know anthropic their view is we need to maximize throughput right maximize throughput but also maximize accuracy

s428
00:41:35.359 --> 00:41:52.319
um you know they want to like you know serve more people um but sometimes it doesn't actually work um sometimes they actually reduce accuracy um and so you can see there is a I don't know if you folks know margin labs um they have this very cool they do they do su bench they benchmark codecs

s429
00:41:52.319 --> 00:41:55.599
they benchmark codecs and clawed code with the models.

s430
00:41:55.599 --> 00:41:59.440
Um, and this is accuracy over time for these models.

s431
00:41:59.440 --> 00:42:03.839
Um, and the dotted lines are the release of the new models.

s432
00:42:03.839 --> 00:42:08.319
Um, so there's actually another um, there was actually a dip in Wait, can you is there Oh, okay.

s433
00:42:08.319 --> 00:42:09.040
The mouse is there.

s434
00:42:09.040 --> 00:42:10.800
Um, I I think it was over here.

s435
00:42:10.800 --> 00:42:13.440
Um, I think it was over here that feeble got released.

s436
00:42:13.440 --> 00:42:15.280
Um, so there was actually another dotted line.

s437
00:42:15.280 --> 00:42:17.920
Um, there was actually very interesting trends you can see.

s438
00:42:17.920 --> 00:42:29.440
The first one is um every single time there is a new model release this this you know daily tracker seems to decrease in accuracy.

s439
00:42:29.440 --> 00:42:38.640
Um and so if you want to predict when a model gets released from anthropic you can use this as an indicator um of when the model gets released.

s440
00:42:38.640 --> 00:42:39.680
it worked very very well.

s441
00:42:39.680 --> 00:42:39.839
Right?

s442
00:42:39.839 --> 00:42:45.839
So like essentially if you were over here the dipped in accuracy over a very long period of time was because fable got released.

s443
00:42:45.839 --> 00:42:49.839
Um and over here I think that's opus 4.8 I think.

s444
00:42:49.839 --> 00:42:55.680
Um I think yeah I think that's opus 4.8. This is opus 4. Uh 7 and so on.

s445
00:42:55.680 --> 00:42:57.280
That's 4.6 I think.

s446
00:42:57.280 --> 00:43:05.040
I whatever um I don't remember exactly but um but you can also see that there is ginormous dips of accuracy.

s447
00:43:05.040 --> 00:43:11.680
Um, and it's not just like one day or two days, it's for a very long period of time.

s448
00:43:12.400 --> 00:43:13.680
This is also Codex.

s449
00:43:13.680 --> 00:43:16.240
Um, so they also do codeex benchmarks.

s450
00:43:16.240 --> 00:43:18.319
Um, and you can also see that over time.

s451
00:43:18.319 --> 00:43:24.400
I don't know if you can squint, but you can see that actually Codeex has been getting worse if you plot the trend, right?

s452
00:43:24.400 --> 00:43:28.400
If you can I don't know if you can squint, but if you draw a line, it seems to be getting worse.

s453
00:43:28.400 --> 00:43:31.440
Um, so I'm assuming OpenAI is investigating this as well.

s454
00:43:31.440 --> 00:43:32.400
Um, okay.

s455
00:43:32.400 --> 00:43:34.400
So this is different model.

s456
00:43:34.400 --> 00:43:36.640
This is codeex.

s457
00:43:37.680 --> 00:43:39.680
So this is using 5.5.

s458
00:43:39.680 --> 00:43:40.480
This is using

s459
00:43:40.480 --> 00:43:42.480
model.

s460
00:43:43.680 --> 00:43:43.920
Correct.

s461
00:43:43.920 --> 00:43:44.800
It's the same.

s462
00:43:44.800 --> 00:43:50.319
So what this benchmark does is you randomly sample 50bench questions.

s463
00:43:50.319 --> 00:43:51.440
Sweet bench is very large.

s464
00:43:51.440 --> 00:43:55.760
So you just sample 50 of them and then you call the model to answer it.

s465
00:43:55.760 --> 00:43:57.839
Um and then you record accuracy.

s466
00:43:57.839 --> 00:44:02.480
Um and so obviously you know every single day there's like you know daily variations.

s467
00:44:02.480 --> 00:44:05.520
Oh, it's not it's not that useful because you're only calling 50 questions.

s468
00:44:05.520 --> 00:44:07.920
Um, so the trick is to look at the trend.

s469
00:44:07.920 --> 00:44:12.240
Um, and the trend uh h maybe open should investigate this.

s470
00:44:12.240 --> 00:44:16.160
Um, and you can see the trend for you know anthropic is also not very good.

s471
00:44:16.160 --> 00:44:20.160
Um, in general oh so sorry this is not the same model.

s472
00:44:20.160 --> 00:44:22.560
Um, these models change my bad.

s473
00:44:22.560 --> 00:44:25.920
Um, so it's the same harness but the model changes.

s474
00:44:25.920 --> 00:44:37.119
Um, so this dotted line is GBD 5.5. Um, so everything over here is GBD 5.5. Everything over here is GBD 5.4. Um I think this is 5.3. Um and so on.

s475
00:44:37.119 --> 00:44:39.839
Um but it seems like the model's getting worse.

s476
00:44:39.839 --> 00:44:41.680
So I don't know.

s477
00:44:41.680 --> 00:44:43.599
This is probably just on this benchmark, right?

s478
00:44:43.599 --> 00:44:47.599
On the Sweet Bench Pro benchmark, it's getting worse.

s479
00:44:47.599 --> 00:44:51.119
Um but you know, I wouldn't really trust these benchmarks.

s480
00:44:51.119 --> 00:44:53.920
The best way is to look at the degradation.

s481
00:44:53.920 --> 00:44:54.960
You know, the sudden drops.

s482
00:44:54.960 --> 00:44:58.000
You know, for example, Codex dramatically dropped over here.

s483
00:44:58.000 --> 00:44:58.880
I don't know why.

s484
00:44:58.880 --> 00:45:08.319
Um and you know clawed you know clawed code was very bad for a few weeks over here or over here right.

s485
00:45:08.720 --> 00:45:09.040
Okay.

s486
00:45:09.040 --> 00:45:11.040
Yes.

s487
00:45:12.240 --> 00:45:14.480
Yes there is a confidence interval.

s488
00:45:14.480 --> 00:45:16.960
I did not plot it but this is 50 tasks.

s489
00:45:16.960 --> 00:45:20.319
So every single day they call a 50 tasks randomly.

s490
00:45:20.319 --> 00:45:22.079
So they will sample 50 tasks.

s491
00:45:22.079 --> 00:45:24.400
Um and so you you should not look at this daily.

s492
00:45:24.400 --> 00:45:25.280
This is daily.

s493
00:45:25.280 --> 00:45:26.880
So every single day is 50 questions.

s494
00:45:26.880 --> 00:45:27.839
Another 50 questions.

s495
00:45:27.839 --> 00:45:29.440
another 50 questions and so on.

s496
00:45:29.440 --> 00:45:34.720
Instead, you should do like a rolling average, you know, some sort of rolling, you know, 7-day average.

s497
00:45:34.720 --> 00:45:36.240
Um, that's a better number.

s498
00:45:36.240 --> 00:45:38.240
Yeah,

s499
00:45:40.880 --> 00:45:41.200
really.

s500
00:45:41.200 --> 00:45:42.160
I can see it from here.

s501
00:45:42.160 --> 00:45:44.480
It's like decreasing.

s502
00:45:46.160 --> 00:45:48.480
It's it's

s503
00:45:52.960 --> 00:45:56.480
if you look at the if you do the seven moving average, I I'll probably get the pot later.

s504
00:45:56.480 --> 00:45:57.359
It actually is decreasing.

s505
00:45:57.359 --> 00:46:06.560
You can see it um if you can see I don't know if you look at the top peaks of the you look at the top peaks and the peaks are decreasing.

s506
00:46:09.920 --> 00:46:12.800
Okay, how about the bottom peaks?

s507
00:46:17.359 --> 00:46:18.800
Okay, I agree there is random noise.

s508
00:46:18.800 --> 00:46:23.280
So the trick is you need to the moving average and if you look at the moving average you can actually see it's decreasing.

s509
00:46:23.280 --> 00:46:25.520
I I'll probably I'll get the plot later.

s510
00:46:25.520 --> 00:46:26.960
Um you can you can search it.

s511
00:46:26.960 --> 00:46:39.599
It's so go to Margin Labs, search in Margin Labs codeex claw code benchmarks and they do show weekly the weekly trend but I'm just saying this is not this is not to say that the model is getting worse.

s512
00:46:39.599 --> 00:46:47.359
This is just to show that accuracy that you know the sudden dips the accuracy of these models can decrease.

s513
00:46:47.359 --> 00:47:00.319
Um and the question is why you know for example why did claude code over a few weeks why did the performance decrease like why that's the fundamental questions.

s514
00:47:03.760 --> 00:47:05.119
So that is one theory.

s515
00:47:05.119 --> 00:47:15.359
A theory is they might have accident you know that before the model release they are doing testing and so they might have like you know act you know some of the

s516
00:47:15.359 --> 00:47:18.640
some of the queries they route to opus 4.8 eight

s517
00:47:18.640 --> 00:47:20.079
or fable or whatever.

s518
00:47:20.079 --> 00:47:23.440
And the problem is they did not.

s519
00:47:23.440 --> 00:47:27.760
So the main question is if you do route to another model, why did the accuracy decrease?

s520
00:47:27.760 --> 00:47:29.119
It should actually get better.

s521
00:47:29.119 --> 00:47:34.240
And so one of the theories is theory one, they forgot to edit the system prompt.

s522
00:47:34.240 --> 00:47:43.359
And so the system prompt for Fable was different, but then they used the wrong system prompt for, you know, for Opus 4.8, and that is why the accuracy decreased.

s523
00:47:43.359 --> 00:47:49.359
Um and then after the model got released the accuracy went back up because they used the correct system prompt.

s524
00:47:49.359 --> 00:47:50.400
That is one theory.

s525
00:47:50.400 --> 00:47:58.400
Um the other theory is the other theory is okay we're actually going to talk about this is it's actually they're doing tricks.

s526
00:47:58.400 --> 00:48:02.160
You know they did quantization but they didn't do dynamic quantization.

s527
00:48:02.160 --> 00:48:03.920
They did some dumb quantization.

s528
00:48:03.920 --> 00:48:07.440
Um you know they some GPUs are broken for example.

s529
00:48:07.440 --> 00:48:08.880
You know they use the wrong GPUs.

s530
00:48:08.880 --> 00:48:10.640
Some of them have like you know bit flips or something.

s531
00:48:10.640 --> 00:48:11.440
I don't know.

s532
00:48:11.440 --> 00:48:17.520
um they have like a new data center and then that data center just by chance has lower accuracy.

s533
00:48:17.520 --> 00:48:21.839
Um in fact there is actually okay I'm going to talk about this actually.

s534
00:48:21.839 --> 00:48:28.559
Um yeah but there are many many theories like you know possibilities why this could reduce an accuracy.

s535
00:48:28.559 --> 00:48:30.880
Um actually I I think it's the next plot.

s536
00:48:30.880 --> 00:48:31.839
Yes the next plot.

s537
00:48:31.839 --> 00:48:34.319
Um oh well the next slides.

s538
00:48:34.319 --> 00:48:37.359
Um so actually when was this?

s539
00:48:37.359 --> 00:48:38.240
I don't remember.

s540
00:48:38.240 --> 00:48:40.640
Um it was a few months ago.

s541
00:48:40.640 --> 00:48:45.359
Someone from AMD actually made an issue on chord code you know during this dip.

s542
00:48:45.359 --> 00:48:54.720
I think it was during the before um a very large dips in accuracy and they actually asked Claude, you know, they asked the Claude team, why is there a noticeable dip in accuracy?

s543
00:48:54.720 --> 00:48:57.920
You know, why why is that?

s544
00:48:57.920 --> 00:49:06.319
And Claude actually wrote a in April 23, they actually provided details on why they had reduced in accuracy, right?

s545
00:49:06.319 --> 00:49:09.599
So they did a postmortem on what happened with Claude.

s546
00:49:09.599 --> 00:49:19.520
Um and the reason why is because the thinking trace got deleted after the second you know when you when you ask call the second time the thinking trace got deleted.

s547
00:49:19.520 --> 00:49:21.920
Um and it had a bad system prompt.

s548
00:49:21.920 --> 00:49:26.400
Um and they found out that that that was why the accuracy got deleted.

s549
00:49:26.400 --> 00:49:35.359
Um so somehow in claude code the second time you ask a question the previous thinking trace got erased.

s550
00:49:35.359 --> 00:49:46.160
Um and I don't know I don't even know how they did not find this but oh wow um according to them now is claude now has this internal benchmark so they will use more internal investigations

s551
00:49:46.160 --> 00:49:55.760
to test okay next time if there's a new model this won't happen ever again um and you know like these things do happen over time um and so like for this specific example

s552
00:49:55.760 --> 00:50:06.240
you know cloud code the harness the harness itself was the problem not the actual model right the harness the thinking trace got deleted and they had a very not a very good system prompt.

s553
00:50:06.240 --> 00:50:10.960
Um and that is why the accuracy actually degra um degraded.

s554
00:50:10.960 --> 00:50:17.200
So that okay so we found one answer why these models got worse.

s555
00:50:19.200 --> 00:50:30.880
They also released in September 2025 right in September 2025 they showed that it was due to okay I didn't okay I didn't put the slide but anyways they showed it was actually due to a hardware problem.

s556
00:50:30.880 --> 00:50:34.880
Um so in their compiler um they used TPUs.

s557
00:50:34.880 --> 00:50:38.319
So so Anthropic likes to use TPUs and GPUs.

s558
00:50:38.319 --> 00:50:45.040
Um they showed that the same software stack for GPUs and TPUs um actually produced different results.

s559
00:50:45.040 --> 00:50:49.280
Um and so for the TPUs it actually was different sampling.

s560
00:50:49.280 --> 00:50:52.160
Um and for the GPUs it was a different sampling mechanism.

s561
00:50:52.160 --> 00:50:56.240
Um and so that is actually why they had decrease in accuracy during September sometime.

s562
00:50:56.240 --> 00:50:58.720
Um because they actually had different hardware.

s563
00:50:58.720 --> 00:51:00.079
And so you need to like Yeah.

s564
00:51:00.079 --> 00:51:05.839
So like once you have different hardware accuracy also changes.

s565
00:51:06.880 --> 00:51:14.400
So I think the main point is the harness the implementation the tool is now the most important.

s566
00:51:14.400 --> 00:51:17.119
It's not the model right the model is useless.

s567
00:51:17.119 --> 00:51:22.960
Most models you know if you look at the model of you know open source versus closed source models are generally the same.

s568
00:51:22.960 --> 00:51:28.960
The difference is how clawed code is made you know how codec is made and used.

s569
00:51:28.960 --> 00:51:31.280
Um and so that is actually the most important factor.

s570
00:51:31.280 --> 00:51:33.040
It's not the model anymore.

s571
00:51:33.040 --> 00:51:43.520
Um and so like you know as we have seen if they have accidentally botched you know if they accidentally botched the harness you will get reduced accuracy.

s572
00:51:43.520 --> 00:51:50.960
Um and so like you know definitely you know for large labs as you know I'm sure they are they know these problems and they're working on it.

s573
00:51:50.960 --> 00:51:54.720
Um but I feel like you know these are probably you know these are still very hard to fix.

s574
00:51:54.720 --> 00:52:07.920
Um yeah so hopefully I answered some of people's questions on the harness you know the accuracy um so there's actually reasons why accuracy got degraded but you know it's not just closource

s575
00:52:07.920 --> 00:52:28.319
labs during bad across open-source model providers the accuracy changes so if you look at this plot so this is from open router um this is deepseek v4 pro um so most labs what they want to do most inference providers what they want to do is they want to serve you the highest throughput,

s576
00:52:28.319 --> 00:52:29.760
right, with the cheapest price.

s577
00:52:29.760 --> 00:52:35.920
They want to give you, you know, 60 tokens, 120 tokens, 1,000 tokens per second, right?

s578
00:52:35.920 --> 00:52:37.359
They want to give you the fastest.

s579
00:52:37.359 --> 00:52:42.000
Um, but did you but did people actually bother to check accuracy?

s580
00:52:42.000 --> 00:52:43.599
So, that is the fundamental question.

s581
00:52:43.599 --> 00:52:48.000
You know, you might be getting 10,000 tokens per second and there is no model.

s582
00:52:48.000 --> 00:52:53.680
Um, so the main question is you need to be careful of what you use from these inference providers.

s583
00:52:53.680 --> 00:53:09.359
Um and so for DeepS v4 you know there are two benchmarks which open router ran you know yeah it's like sorted it's sorted I think on the gray I think it's sorted on tower bench um so it's sorted on tower bench um and the green one is GPQA

s584
00:53:09.359 --> 00:53:16.960
and you can see that in general some of the labs are not you know some of the sorry not labs some of the inference providers are not doing very well

s585
00:53:16.960 --> 00:53:24.800
um so you need to like before you before you use a open source model please check the accuracy before you use the open source model.

s586
00:53:24.800 --> 00:53:36.559
Um and also one of the biggest problems of this is every single time you know for example a cloud code you know claude code and codeex you can benchmark accuracy over time.

s587
00:53:36.559 --> 00:53:41.520
Um and the good thing about closed source labs is they control the supply chain.

s588
00:53:41.520 --> 00:53:56.240
Um the biggest problem of open source is there are so many suppliers and providers of these models um that sometimes what happens is people get turned off and they get very annoyed that the open source models do not work very well.

s589
00:53:56.240 --> 00:54:13.520
Um so everyone you know in the in the ecosystem people keep saying that closed source labs do much better than open source but it's not because of the model it's because of the inference provider right the inference provider is to blame that they are causing the downfall of open source because they're giving a bad name for open

s590
00:54:13.520 --> 00:54:28.160
source um so I would like check you know whatever favorite inference provider you have um so this this benchmark was run I think yesterday um by open router so this is this is daily data by open router.

s591
00:54:28.160 --> 00:54:36.400
Um so whatever favorite inference provider you have please tell them not to you know reduce accuracy that much.

s592
00:54:36.400 --> 00:54:43.040
Um this is GLM 5.2. Um so you know GLM 5.2 as well shows different accuracies.

s593
00:54:43.040 --> 00:54:53.920
Um you can see so the plot on the right shows most model you know most inference provide okay I keep saying model apps most inference providers are throughput maxing

s594
00:54:53.920 --> 00:55:04.720
but they are accuracy minimizing that's where the phrase comes from okay so they do not care about in fact look like you know the highest accuracy is 76.4% and the lowest is 62.4%.

s595
00:55:04.720 --> 00:55:10.880
So there is a 10% gap between the back you know between the highest accuracy and the you know lowest accuracy.

s596
00:55:10.880 --> 00:55:24.880
Um and so like you need to you know as a as a you know as a call out to inference providers you know please increase accuracy you know before trying to make things faster right you do not want a model to be very dumb

s597
00:55:24.880 --> 00:55:36.079
um and it's like you know 10,000 tokens per second right we can make it 1 million tokens per second and there is no model um you know just call a human or something you know make a fake or something so

s598
00:55:36.079 --> 00:55:47.440
yeah so the main point is we need inference providers to do good in terms of accuracy otherwise this will make open source have a very bad look um yeah

s599
00:55:47.440 --> 00:55:55.280
oh okay that's the end of the the second section I guess that was a bit of a rant um any other questions for this yes

s600
00:55:55.280 --> 00:56:12.319
for a new organization that's that wants to use like the open source model do you suggest using a you know inference service provider or do you suggest downloading from hugging face and then using like model or you know some kind server to you know implement

s601
00:56:12.319 --> 00:56:18.559
yourself like what do you suggest if any new organization comes and asks you like how do you use open source model

s602
00:56:18.559 --> 00:56:27.520
that's a great question so when an open source model gets released you know how should you use it in terms of accuracy throughput or whatever um so in general

s603
00:56:27.520 --> 00:56:39.680
um in general open source has come a long way so for example we did report bugs in Jamaa 1 ja 2 llama mistro you know open gdosss every single of those models had bugs Um

s604
00:56:39.680 --> 00:56:46.480
and so the good thing is you know as we will help the labs before they release a model to fix some of the issues.

s605
00:56:46.480 --> 00:56:50.079
So every single model you now have has some of our fixes.

s606
00:56:50.079 --> 00:56:51.040
So that's a good thing.

s607
00:56:51.040 --> 00:56:57.359
Um but in general if you have a open source model I would use Llama CPP for example.

s608
00:56:57.359 --> 00:57:03.040
I think Llama CPP and Llama server is probably the most bugfree system.

s609
00:57:03.040 --> 00:57:06.480
So I would like suggest yes you should download from hugging face.

s610
00:57:06.480 --> 00:57:14.000
use Llama server, use Llama, you know, CLI, I don't know, you can use Unsoft Studio, whatever, whatever is your favorite tool.

s611
00:57:14.000 --> 00:57:16.559
But you should, yes, you should download from Hugging Base.

s612
00:57:16.559 --> 00:57:23.680
Um, in terms of like, you know, if you're a large enterprise, generally speaking, what they like to do is they like to wait one week.

s613
00:57:23.680 --> 00:57:28.319
So most enterprises, they'll wait one week for all the problems to be fixed.

s614
00:57:28.319 --> 00:57:31.280
Um, and then, you know, then they will use the model.

s615
00:57:31.280 --> 00:57:33.520
But in my view, that is not a good approach.

s616
00:57:33.520 --> 00:57:39.280
I would say if you okay if everyone waits one week then like how do we fix the bugs?

s617
00:57:39.280 --> 00:57:43.839
Um because only at scale only at scale then we can see the bugs.

s618
00:57:43.839 --> 00:57:49.200
Um and so like in general we need everyone to start trying these models earlier.

s619
00:57:49.200 --> 00:57:54.559
Um and not like you know wait one week wait one month you know don't do don't do the waiting approach.

s620
00:57:54.559 --> 00:57:59.920
Um but I would say like in general the enterprises what they like to do is just wait one week.

s621
00:57:59.920 --> 00:58:01.839
Um yeah that's like common practice.

s622
00:58:01.839 --> 00:58:04.559
Um, yes.

s623
00:58:14.640 --> 00:58:19.680
So, okay, the question was why would the model performance degrade before a model release?

s624
00:58:19.680 --> 00:58:25.599
Um, these are just hypothetical question hypothetical theories.

s625
00:58:25.599 --> 00:58:29.520
So, every single model has a different system prompt.

s626
00:58:29.520 --> 00:58:33.200
So, opus 4.8 8 Opus 4.8 system prompt is very short.

s627
00:58:33.200 --> 00:58:36.960
Um but Opus 4.7 system prompter was extremely long.

s628
00:58:36.960 --> 00:58:51.440
Um so the theory was this is just a theory that anthropic via claude code accidentally routed some of the models to Opus 4.8 right they use opus 4.8 as testing right they need to test opus 4.8

s629
00:58:51.440 --> 00:58:54.319
But they used Opus 4.7 system prompt.

s630
00:58:54.319 --> 00:58:58.160
So they used the wrong system prompt and that is why accuracy degraded.

s631
00:58:58.160 --> 00:58:59.200
That's one theory.

s632
00:58:59.200 --> 00:59:04.799
Um, another theory is actually I think that's the actually I thought about it.

s633
00:59:04.799 --> 00:59:06.400
That's probably the only theory I had.

s634
00:59:06.400 --> 00:59:08.640
I'm like thinking hm is there another theory?

s635
00:59:08.640 --> 00:59:10.640
Um,

s636
00:59:12.400 --> 00:59:25.920
I guess the harness itself like you know sometimes the harness itself the harness was was designed for Opus 4.7. Um and during when they were going to release 4.8,

s637
00:59:25.920 --> 00:59:28.319
they need to collaborate the harness, right?

s638
00:59:28.319 --> 00:59:32.079
They need to change the harness um for 4 4.8 to make it work.

s639
00:59:32.079 --> 00:59:36.000
But the problem is you're not allowed to publish it, right?

s640
00:59:36.000 --> 00:59:42.240
You're not allowed to publish it and give it to people because otherwise people will like, you know, go into Twitter on LinkedIn, you know, everywhere.

s641
00:59:42.240 --> 00:59:44.799
Oh, I can see 4.8 is going to be released.

s642
00:59:44.799 --> 00:59:47.599
You know, everyone's going to be screaming, you know, 4.8's coming.

s643
00:59:47.599 --> 00:59:51.839
4.8's, eights, you know, were getting released and so maybe that's why accuracy decreased.

s644
00:59:51.839 --> 00:59:54.559
It's they update they did not update the harness.

s645
00:59:54.559 --> 01:00:05.520
Um or the other option is they up they already up they silently they silently updated the harness before the new model got released and it regressed you know it reduced accuracy.

s646
01:00:05.520 --> 01:00:09.680
Um I don't know like to be honest I you should probably ask anthropic that question.

s647
01:00:09.680 --> 01:00:16.640
Um or but I think in general in general the dips the dips don't always correspond to like new model releases.

s648
01:00:16.640 --> 01:00:27.359
Some of the dips are actual issues like you know the thinking trace got deleted um the system prompt they wrote a wrong I think for the system it's funny I think for the system prompt they said

s649
01:00:27.359 --> 01:00:46.400
um they tried to reduce verbosity so they tried to make the model less talkative um and it actually made the model dumber um and so I think it was just one word they added one word no one sentence I think one sentence in the system prompt that made the model dumber Um,

s650
01:00:46.400 --> 01:00:50.160
yeah, I don't know if that helps, but I don't know if anyone else has any like theory.

s651
01:00:50.160 --> 01:00:53.280
I don't I don't think so anyone even has that many theories on this.

s652
01:00:53.280 --> 01:00:57.599
Um, obviously the anthropic engineers will know, but I, you know, they're not going to tell.

s653
01:00:57.599 --> 01:01:00.240
So, it's just based on hypotheticals.

s654
01:01:00.240 --> 01:01:03.599
Something to do with the system problems, something to do with the harness.

s655
01:01:03.599 --> 01:01:03.920
Yeah.

s656
01:01:03.920 --> 01:01:05.760
But I think in general, you can also use this plot.

s657
01:01:05.760 --> 01:01:09.280
You know, if the performance decreases, most likely a new model is going to be coming.

s658
01:01:09.280 --> 01:01:10.480
Um, yeah.

s659
01:01:10.480 --> 01:01:11.520
Any other qu?

s660
01:01:11.520 --> 01:01:12.000
Yes.

s661
01:01:12.000 --> 01:01:21.200
Just to add on that Yes, correct.

s662
01:01:26.240 --> 01:01:28.240
Correct.

s663
01:01:32.480 --> 01:01:32.799
Yes.

s664
01:01:32.799 --> 01:01:33.839
Exactly.

s665
01:01:33.839 --> 01:01:34.480
Exactly.

s666
01:01:34.480 --> 01:01:40.160
So before a model release, they use a different system prompt for that new model for the old model.

s667
01:01:40.160 --> 01:01:42.720
And so that is probably why there are some decrease in accuracy.

s668
01:01:42.720 --> 01:01:45.119
they switch the system prompts around or something like that.

s669
01:01:45.119 --> 01:01:51.280
Um and also you know the model itself you know I think 4.8 system prompts is very short.

s670
01:01:51.280 --> 01:01:55.599
Um it's yeah I think it's like very very short and 4.7 was ginormous.

s671
01:01:55.599 --> 01:02:02.720
Um and the reason is 4.7 was like you know I don't know what I don't know what happened but they have this ginormous system prompt and the 4.8 just shrunk it a lot.

s672
01:02:02.720 --> 01:02:12.480
Um so maybe maybe they used the 4.7 system prompt I don't know or 4.8 system the short system prompt for 4.8 eight and then they use it for 4.7

s673
01:02:12.480 --> 01:02:14.160
and that's why it decreased accuracy.

s674
01:02:14.160 --> 01:02:14.720
I don't know.

s675
01:02:14.720 --> 01:02:16.000
Um but yeah, you're correct.

s676
01:02:16.000 --> 01:02:21.359
Um they do release although I think the system prompt they released on the website is for claw.ai.

s677
01:02:21.359 --> 01:02:26.240
So the online chat system um the clawed code system prompt is actually different.

s678
01:02:26.240 --> 01:02:26.799
Yeah.

s679
01:02:26.799 --> 01:02:35.839
So I think you need to actually call you need to call claude code you know what is my system prompt and then you print it to like a text file

s680
01:02:35.839 --> 01:02:47.119
um and then you can like in investigate what the system prompt is and then you can also override it if you want um yes but it's a different system prompt most likely um yeah

s681
01:02:47.119 --> 01:03:03.440
last question if anyone no okay continue on Okay, the next section we're going to be talking about is benchmaxing and cheating.

s682
01:03:03.440 --> 01:03:09.920
Um, I'm not sure if you folks have seen the deep SWE benchmark.

s683
01:03:09.920 --> 01:03:21.039
Um, the deep SWE benchmark is a very popular recent benchmark that shows, you know, the cost is on the X-axis and the Y-axis it is a deep SWE benchmark.

s684
01:03:21.039 --> 01:03:26.880
It's a new benchmark based on like you know a better uncontaminated version of Swebench Pro.

s685
01:03:26.880 --> 01:03:43.599
Um and in general you can see that you know GBD 5.5 does very well with Fable um you know GLM Opus 4.8 in general right it shows you know this plot shows that models are getting you know um these the dots are different reasoning modes um I think this is

s686
01:03:43.599 --> 01:03:54.400
maximum reasoning I think um high extra high you know these are actually different reasoning um reasoning times as well um but in general you can see that there is a parto

s687
01:03:54.400 --> 01:04:02.960
efficiency trend right the best model is the one you know to the right to the top right the better the model to the right to the top is the better the model.

s688
01:04:02.960 --> 01:04:08.640
Um, so you want the models to do better and better over time to that to the top right corner.

s689
01:04:09.280 --> 01:04:12.400
And you know, I just learned I didn't actually know this.

s690
01:04:12.400 --> 01:04:21.359
I just learned that Sweetbench Pro when you run this benchmark you use LL you use language models as a verifier.

s691
01:04:21.359 --> 01:04:33.920
Um, and I was like confused because like for most benchmarks, for most benchmarks, you should never call another language model to check whether your answer is right or wrong.

s692
01:04:33.920 --> 01:04:40.720
And so for Swedebench Pro, you actually call a language model to verify if your language model was right.

s693
01:04:40.720 --> 01:04:45.200
Um, and so that is why SweetBench Pro is not a very good benchmark.

s694
01:04:45.200 --> 01:04:51.039
Um, one of the problems is is do we need to do sampling?

s695
01:04:51.039 --> 01:04:56.160
Like how many verification runs do you need to run to verify if your answer is correct?

s696
01:04:56.160 --> 01:04:58.079
Do you run it one time?

s697
01:04:58.079 --> 01:04:59.520
Do you run it five times?

s698
01:04:59.520 --> 01:05:01.839
You run it 100 times and take like an average.

s699
01:05:01.839 --> 01:05:05.520
Um, so I was actually quite shocked that this is actually what happens.

s700
01:05:05.520 --> 01:05:07.039
I was quite surprised actually.

s701
01:05:07.039 --> 01:05:10.240
Um, the next question is which model is the verifier?

s702
01:05:10.240 --> 01:05:25.920
You know you ask for example you ask opus you know you you benchmark opus 4.8 on swbench pro but which what do you use as a verifier do you use opus 4.8 eight as the verifier to using the same model itself to verify itself.

s703
01:05:25.920 --> 01:05:31.119
Um and so like this I was like quite surprised actually that this is how benchmarks work.

s704
01:05:31.119 --> 01:05:33.359
Um and actually quite disappointed.

s705
01:05:33.359 --> 01:05:37.359
Um but anyways obviously you can go the other approach.

s706
01:05:37.359 --> 01:05:39.440
You can do human verification.

s707
01:05:39.440 --> 01:05:45.839
Um you know everyone in the room I'll give you the bench you know and just tell you guys to verify it.

s708
01:05:45.839 --> 01:05:47.599
Um you could do that I guess.

s709
01:05:47.599 --> 01:05:50.720
Um and also what happens if the verification changes every day?

s710
01:05:50.720 --> 01:05:57.359
Um you know remember previously models you know every single day models get better or worse.

s711
01:05:57.359 --> 01:06:07.280
um what happens what happens if you run what happens if you run the verification when the model was doing very bad right you will actually have different SWE numbers

s712
01:06:07.280 --> 01:06:19.280
um and so like I'm actually quite surprised this is what the industry does um you know run bench pro but using anonymous is verifies that is definitely not a good idea um but anyways people do it whatever

s713
01:06:19.280 --> 01:06:31.760
um in fact according to deepu um if you do if you do verification using language models Sweet Bench Pro has a 8.5% false positive rate.

s714
01:06:31.760 --> 01:06:39.359
Um, and a false positive rate means that the LLM verifier said that the model was correct, but it was actually wrong.

s715
01:06:39.359 --> 01:06:43.280
Um, and so 8.5% of the time it would do this.

s716
01:06:43.280 --> 01:06:48.559
Um, the false negative rate is even worse at 24%.

s717
01:06:48.559 --> 01:06:54.480
Um, this means that the verifier said that the model was wrong, but it was actually right.

s718
01:06:54.480 --> 01:06:58.720
Um and so you can see that SweetBench Pro is a very bad benchmark.

s719
01:06:58.720 --> 01:07:10.240
Um and so Deep Sweet showed that they have in you know they fixed the problem you know um by reducing the false positive rate and the false negative rate to you know 1%.

s720
01:07:10.240 --> 01:07:25.359
Um in fact some examples of cheating um I you know this is actually quite surprising um but in the bench pro benchmark you get you get like a GitHub question you know a GitHub issue

s721
01:07:25.359 --> 01:07:37.119
you call the model to solve that GitHub issue but did you know that in Swebench Pro you get the full Git history so you get the you get the actual answer as well

s722
01:07:37.119 --> 01:07:47.839
um so I'm like I'm actually quite I was actually quite shocked to learn this um that during these models you give the answer and the question like obviously the model will cheat.

s723
01:07:47.839 --> 01:07:51.359
Um and so like this is definitely a very bad benchmark.

s724
01:07:51.359 --> 01:07:56.799
Um you know you should never ever ever ever give the model the answer.

s725
01:07:56.799 --> 01:07:59.440
Um and so very silly.

s726
01:07:59.440 --> 01:08:01.440
Um but yes this happens a lot.

s727
01:08:01.440 --> 01:08:04.960
Um and you do not want the model to literally see the solution, right?

s728
01:08:04.960 --> 01:08:06.640
That is a terrible approach.

s729
01:08:06.640 --> 01:08:17.679
Um the other problems that you get get like false positives is you know the PR tests you know the the GitHub the GitHub issue tests are very weak.

s730
01:08:17.679 --> 01:08:26.799
Um so you know at the final conclusion you know when the GitHub when the GitHub issue is closed with a pull request the tests that the maintainer wrote are not very good.

s731
01:08:26.799 --> 01:08:33.759
Um and so the problem of that is you know if you have tests which are very weak then you know the model does very well not very good.

s732
01:08:33.759 --> 01:08:37.920
Um and obviously the worst part is the model will like bypass some tests.

s733
01:08:37.920 --> 01:08:42.080
It will skip some um and that is not a very good approach.

s734
01:08:42.080 --> 01:08:52.640
In fact um deep suite actually showed how many times a model cheats by looking at the full git history you know directly going to the answer.

s735
01:08:52.640 --> 01:08:59.440
Um you can see opus 4.7. So the purple bars show cheating by models.

s736
01:08:59.440 --> 01:09:02.719
Um ah it looks like Jubilee 5.5 never cheats.

s737
01:09:02.719 --> 01:09:09.279
It looks like it um h okay maybe we should use GP 5.5. Um you know actually this is actually very interesting.

s738
01:09:09.279 --> 01:09:14.480
There are some people which think that if you cheat that's actually good.

s739
01:09:14.480 --> 01:09:26.080
Um and the reason why it's good is it means that Opus 4.7 already know like if you give it the full Git history you should be able to like you gave it to them right you gave Opus the full Git history

s740
01:09:26.080 --> 01:09:29.839
it should find the solution there right it should just directly skip over to the solution.

s741
01:09:29.839 --> 01:09:43.600
So it's that's what people think you know people have a view that the humans gave Opus 4.7 the full gate history so it should cheat right you you you designed it to cheat um so in general

s742
01:09:43.600 --> 01:10:02.159
cord models seem to cheat more um and open AI models seem to cheat less in general um so it depends on you you know if you want a model to cheat or not um and the definition of the word cheat is also very you know charged so I guess it depends on what the word cheating means

s743
01:10:02.159 --> 01:10:15.280
um you know for false negatives remember Swebench Pro calls a language model to verify if your answer is correct um and so sometimes it's not very good you know sometimes you have unrelated

s744
01:10:15.280 --> 01:10:27.120
tests that fail um you forgot you know sometimes when you write tests you forgot about the tests which have helpers you know helper functions and you just skip that um so there are many issues and this

s745
01:10:27.120 --> 01:10:37.440
I think this was 20 yeah so 24% % of the time, 24% of the time, the model says, the verifier says your model was wrong, but it was actually correct.

s746
01:10:37.440 --> 01:10:40.320
So, this is another problem.

s747
01:10:40.719 --> 01:10:45.520
And even worse, the harness itself can change accuracy.

s748
01:10:45.520 --> 01:10:53.840
So, when you benchmark using Swebench Pro, like you need to have one agent or one harness for all models, right?

s749
01:10:53.840 --> 01:10:58.080
How do you create a generalized control environment for these models?

s750
01:10:58.080 --> 01:11:08.719
Um and so you can see like you know for example DeepSu showed if you use clawed code you get 40% accuracy but then if you use their own so it's a special harness

s751
01:11:08.719 --> 01:11:19.840
you can get 50% accuracy um Gemini for example right if you use Gemini CLI you get 20% accuracy but if you use their one you know the the control environment

s752
01:11:19.840 --> 01:11:29.520
you can get 40% accuracy um and so in general for these benchmarks you also need to have a controlled environment.

s753
01:11:29.520 --> 01:11:32.800
Um, and that is also another problem.

s754
01:11:32.800 --> 01:11:44.560
And with Deep Suite, they showed by using this benchmark, by solving, you know, by stopping cheating, you know, by, you know, if we remove cheating, if we remove, you know, these other issues,

s755
01:11:44.560 --> 01:11:49.520
you can see the models, you know, the models are not saturated anymore, right?

s756
01:11:49.520 --> 01:11:52.960
You can see the models are very different in terms of the capabilities.

s757
01:11:52.960 --> 01:11:56.480
According to this benchmark, GBD 5.5 is the best according to this one.

s758
01:11:56.480 --> 01:11:58.239
Um I don't Oh, this is not updated.

s759
01:11:58.239 --> 01:12:00.960
Um for 4.8 I think is over here or something.

s760
01:12:00.960 --> 01:12:06.000
Um but yes, this benchmark shows core taiku is 0%.

s761
01:12:06.000 --> 01:12:07.679
Um accuracy, right?

s762
01:12:07.679 --> 01:12:09.520
It's terrible, I guess.

s763
01:12:09.520 --> 01:12:13.600
Um but yeah, this benchmark just show okay the main question is do you trust this benchmark?

s764
01:12:13.600 --> 01:12:14.719
That is another question.

s765
01:12:14.719 --> 01:12:18.640
Um there is other benchmarks, right?

s766
01:12:18.640 --> 01:12:25.920
So cognition released a frontier code benchmark which also tries to solve the same questions for benchm you know for cheating and benchmarks.

s767
01:12:25.920 --> 01:12:29.360
Um and what they showed is you can fix contamination.

s768
01:12:29.360 --> 01:12:31.600
And how do you fix contamination?

s769
01:12:31.600 --> 01:12:38.719
You ask, you know, you ask Cognition's team, which is full of like, you know, national Olympiads and, you know, international Olympiads.

s770
01:12:38.719 --> 01:12:46.640
They manually checked every single question um themselves, you know, and removed bad questions, you know, bad examples.

s771
01:12:46.640 --> 01:12:50.480
Um and they also showed that their questions are much more diverse, right?

s772
01:12:50.480 --> 01:12:53.440
So, Frontier Code has many different other languages.

s773
01:12:53.440 --> 01:13:08.800
Um and they showed with diversity you know with more diverse programming languages um and by reducing contamination they also have a benchmark um and according to their benchmark opus 4.8 is the best

s774
01:13:08.800 --> 01:13:13.840
right for 14.5% accuracy juby 5.5 is 7.2 to accuracy.

s775
01:13:13.840 --> 01:13:17.280
Um, and this is the diamond one, right?

s776
01:13:17.280 --> 01:13:20.640
So, this is the 50 the 50 hardest questions.

s777
01:13:20.640 --> 01:13:24.719
Um, the main benchmark is 100 questions and the extended is 150.

s778
01:13:24.719 --> 01:13:32.480
Um, and so according to them, you know, Claude does the best according to them.

s779
01:13:32.480 --> 01:13:38.239
But also according to them, Frontier code seems to be better than Deep Suite.

s780
01:13:38.239 --> 01:13:38.480
Right?

s781
01:13:38.480 --> 01:13:48.239
The benchmark that I showed previously, Deepswuite, this one um you know according to Frontier Code, so the cognition team, their benchmark is better than Deepswuite, right?

s782
01:13:48.239 --> 01:13:54.880
According to them, according to them, Deep SW's false positive rate is 44.9%.

s783
01:13:54.880 --> 01:13:56.800
But remember what did Deepswu say?

s784
01:13:56.800 --> 01:14:02.080
They said the false positive rate was I don't remember what what did they say?

s785
01:14:02.080 --> 01:14:06.239
Um they said that it was 0.3%.

s786
01:14:06.239 --> 01:14:06.560
Right?

s787
01:14:06.560 --> 01:14:11.280
So deep said deep said their false positive rate is 0.3%.

s788
01:14:11.280 --> 01:14:16.640
But Frontier code said that Deep SWE's false positive rate was 44.9%.

s789
01:14:16.640 --> 01:14:31.199
Um so you know there is some competition I guess between benchmarking labs um well cognition is not a benchmarking lab but like you know between companies um so the main question is who do we trust you know do we trust

s790
01:14:31.199 --> 01:14:33.440
Frontier codes benchmarks?

s791
01:14:33.440 --> 01:14:36.560
Do we trust Deep Swiss benchmarks?

s792
01:14:36.560 --> 01:14:38.239
Do we trust Bench?

s793
01:14:38.239 --> 01:14:39.760
You know, who do we trust?

s794
01:14:39.760 --> 01:14:42.159
And that is a very important question.

s795
01:14:42.159 --> 01:14:47.679
Um, you know, my take is like, you know, like let's just take an average of everyone.

s796
01:14:47.679 --> 01:14:50.159
Take an average of everyone and you'll probably get the best answer.

s797
01:14:50.159 --> 01:14:51.679
You know, who is actually doing the best.

s798
01:14:51.679 --> 01:14:54.080
Um, yeah, but this is actually very interesting.

s799
01:14:54.080 --> 01:14:58.480
Um, you know, it show Okay, so according to them, the false negative rate for Deep Suite is correct.

s800
01:14:58.480 --> 01:14:59.600
You know, 1.2%.

s801
01:14:59.600 --> 01:15:11.360
But my interest, you know, I probably, you know, my main question is why is the false positive rate so high for deep according to according to Frontier Bench, Deep Sweet is even worse

s802
01:15:11.360 --> 01:15:12.800
than Sweet Bench Pro.

s803
01:15:12.800 --> 01:15:16.080
That's what they're trying to say, I guess, for for the false positive rate.

s804
01:15:16.080 --> 01:15:18.560
Um, yeah.

s805
01:15:19.760 --> 01:15:23.840
And even worse, there is another benchmark called Frontier Math.

s806
01:15:23.840 --> 01:15:27.040
Um, so Frontier Math is by Epoch AI.

s807
01:15:27.040 --> 01:15:34.080
Um so they they have this math benchmark with different tiers you know tier one, tier two, tier three, tier four.

s808
01:15:34.080 --> 01:15:35.440
So tier four is the hardest.

s809
01:15:35.440 --> 01:15:39.199
Um but the benchmark itself was botched.

s810
01:15:39.199 --> 01:15:44.080
Um and so they actually had to release a corrected version of their benchmark.

s811
01:15:44.080 --> 01:15:48.960
Um I think this was one month ago um or something.

s812
01:15:48.960 --> 01:15:54.080
Um so they showed that their benchmark questions were fully wrong.

s813
01:15:54.080 --> 01:16:02.800
Um, and you can see that if you correct the benchmark, if you correct the benchmark, the accuracy for GBD 5.5 jumps from 50% to 80% or something.

s814
01:16:02.800 --> 01:16:06.560
Um, and so now you kind of trust the benchmark.

s815
01:16:06.560 --> 01:16:09.920
And they showed in a tweet, oh, it's June 12.

s816
01:16:09.920 --> 01:16:11.280
Oh, it's only two weeks ago.

s817
01:16:11.280 --> 01:16:18.880
Um, so in June 12, they showed that the reason why they did bad on the benchmarks is they they did the answer extraction incorrectly.

s818
01:16:18.880 --> 01:16:21.840
For example, they did, you know, they had unclear questions.

s819
01:16:21.840 --> 01:16:23.280
They had the incorrect sign.

s820
01:16:23.280 --> 01:16:29.920
So, for example, they said the model said 12, but it should be actually minus 12 and they forgot to cut the minus sign.

s821
01:16:29.920 --> 01:16:32.560
Um, they have one-off errors.

s822
01:16:32.560 --> 01:16:35.679
Um, yeah, there's many problems with the benchmark.

s823
01:16:35.679 --> 01:16:40.640
Um, and so they fixed their benchmark um, just recently.

s824
01:16:41.679 --> 01:16:44.000
In fact, you know, it's actually quite funny.

s825
01:16:44.000 --> 01:16:47.120
This was just two weeks ago.

s826
01:16:47.120 --> 01:16:52.960
Have you guys heard of hugging faces math verify which was one year ago?

s827
01:16:52.960 --> 01:17:04.080
Um and hugging base showed that in fact these benchmarks when you do math questions they always do bad and the reason why is because there's many problems right the formatting is incorrect

s828
01:17:04.080 --> 01:17:16.960
um you know the extraction of the fraction is wrong um you know the sign is failed extraction there's many many many problems of mathematical extraction and to be honest I feel like it's like kind of a reinventing the wheel or you know

s829
01:17:16.960 --> 01:17:23.600
rediscovery um but hugging base actually published this one year ago and epoch just fixed it 2 weeks ago.

s830
01:17:23.600 --> 01:17:31.679
Um so you know benchmarking labs definitely need more you know they need to investigate literature more I think.

s831
01:17:31.679 --> 01:17:45.360
Um in fact according to hugging face math verify you know if you use if you the green bar the green bar is if you do not use hugging faces's verification system you know to fix the benchmark.

s832
01:17:45.360 --> 01:17:49.520
If you do fix the benchmark, you can see accuracy dramatically increases, right?

s833
01:17:49.520 --> 01:17:54.400
For example, for Quen, for Quen, the accuracy was 10%, now it's 25%.

s834
01:17:54.400 --> 01:17:58.480
Um, and so you need to, so that means the open source models are not dumb.

s835
01:17:58.480 --> 01:18:01.760
They just have different they output a different format.

s836
01:18:01.760 --> 01:18:08.640
Um, and so one of the problems is how do we actually actually like, you know, pass these different formats?

s837
01:18:08.800 --> 01:18:10.159
In fact, it's even worse.

s838
01:18:10.159 --> 01:18:15.120
Um, no, I think I tweeted, oh, I tweeted this in August 2024.

s839
01:18:15.120 --> 01:18:20.000
Um, that if you if you use different tokenization, you can also have different accuracy.

s840
01:18:20.000 --> 01:18:26.640
Um, in fact, for MLOU, if you use spaces, you increase accuracy by 0.4%.

s841
01:18:26.640 --> 01:18:37.679
Um, it might not sound like a lot, but the point is by these very dumb things like, you know, using spaces or, you know, minus 12 becomes 12.

s842
01:18:37.679 --> 01:18:44.320
Um and all of these like dumb little small things, the accuracy of these benchmarks can change over time.

s843
01:18:44.320 --> 01:18:56.320
Um and so like the main question is you know how do we make benchmarking labs and benchmarking companies you know how do we make them more reliable um and you know more trustworthy.

s844
01:18:57.199 --> 01:19:00.719
Oh okay that's I guess the section for the benchmarking part.

s845
01:19:00.719 --> 01:19:02.960
Any other questions for that section?

s846
01:19:02.960 --> 01:19:05.760
Um questions.

s847
01:19:05.760 --> 01:19:07.760
Yes.

s848
01:19:40.880 --> 01:19:41.679
That's a great question.

s849
01:19:41.679 --> 01:19:50.480
So the question is how do we how can we trust these benchmarking companies or like what other types of benchmarks can we do to make it trustworthy?

s850
01:19:50.480 --> 01:19:52.640
Um so that is actually a very good question.

s851
01:19:52.640 --> 01:19:56.400
The main question for benchmarks is you need to satisfy two conditions.

s852
01:19:56.400 --> 01:20:01.040
The first condition is the benchmark must not must not be benchmaxable.

s853
01:20:01.040 --> 01:20:01.440
Right?

s854
01:20:01.440 --> 01:20:05.920
How do you make a benchmark that is extremely hard to benchmark, right?

s855
01:20:05.920 --> 01:20:08.400
How do we like not get 100% accuracy?

s856
01:20:08.400 --> 01:20:13.120
And the second question is how do we make the benchmark um verifiable, right?

s857
01:20:13.120 --> 01:20:18.080
So how do we make the benchmark you can you can also verify that the answer is in fact correct, right?

s858
01:20:18.080 --> 01:20:25.440
You remember Swebench Pro is dumb because you call the language model itself to verify itself.

s859
01:20:25.440 --> 01:20:26.880
Um so that is not good.

s860
01:20:26.880 --> 01:20:29.360
Um so the main question is those two questions.

s861
01:20:29.360 --> 01:20:34.880
Um, and so one good example, this is just a dumb example.

s862
01:20:36.239 --> 01:20:39.679
Randomly create maths questions.

s863
01:20:39.679 --> 01:20:40.960
Sample for example.

s864
01:20:40.960 --> 01:20:44.320
Okay, this is okay, this is probably not a good benchmark.

s865
01:20:44.320 --> 01:20:46.719
You automatically create maths questions.

s866
01:20:46.719 --> 01:20:49.440
Um, we can sample infinity, right?

s867
01:20:49.440 --> 01:20:51.840
We can sample infinite maths questions, right?

s868
01:20:51.840 --> 01:20:56.560
2+ 2, 4 plus 4, you know, any single number added together.

s869
01:20:56.560 --> 01:20:58.480
That's one question.

s870
01:20:58.480 --> 01:20:59.679
Can you verify this?

s871
01:20:59.679 --> 01:21:00.320
Yes, you can.

s872
01:21:00.320 --> 01:21:00.640
Right?

s873
01:21:00.640 --> 01:21:04.640
You can call a calculator to verify what is 2 plus two.

s874
01:21:04.640 --> 01:21:07.040
Can this be benchmaxible?

s875
01:21:07.040 --> 01:21:08.159
Hard.

s876
01:21:08.159 --> 01:21:11.760
And the reason why hard is because the sampling space is infinity, right?

s877
01:21:11.760 --> 01:21:14.960
It can be 2 plus 2, 1,00 plus 101, right?

s878
01:21:14.960 --> 01:21:17.040
It can you you don't have to do plus, right?

s879
01:21:17.040 --> 01:21:19.280
You can do 1,00 times 1,00.

s880
01:21:19.280 --> 01:21:26.560
And so that's one way make a benchmark which is very hard to cheat but also easy to verify.

s881
01:21:26.560 --> 01:21:27.760
So some sort of math question.

s882
01:21:27.760 --> 01:21:33.040
Um, the other one, for example, is um, okay, maybe this is not a good example.

s883
01:21:33.040 --> 01:21:35.840
I'm just making this one up on the spot.

s884
01:21:35.840 --> 01:21:44.480
Tell the model to create a poem in 70 words and you must use the word happy.

s885
01:21:44.480 --> 01:21:45.600
Can you verify this?

s886
01:21:45.600 --> 01:21:46.640
Yes, you can.

s887
01:21:46.640 --> 01:21:49.520
Is happy in the, you know, generation?

s888
01:21:49.520 --> 01:21:51.440
If yes, plus one.

s889
01:21:51.440 --> 01:21:53.360
Also, you can count how many words, right?

s890
01:21:53.360 --> 01:21:55.360
You can count, okay, is there 70 words?

s891
01:21:55.360 --> 01:21:58.400
Um, so you can do these type of approaches.

s892
01:21:58.400 --> 01:21:59.840
And is this benchmaxable?

s893
01:21:59.840 --> 01:22:09.440
No. It's it's very hard to benchmark because you can say 70 words, 69 words, 68 words, 102 words, 1,000 words, right?

s894
01:22:09.440 --> 01:22:10.480
It doesn't have to be happy.

s895
01:22:10.480 --> 01:22:13.920
It can be you must have two words, you must have three words.

s896
01:22:13.920 --> 01:22:19.760
Um, so some some sort of benchmark where it's very hard to benchmark.

s897
01:22:19.760 --> 01:22:28.159
Um yeah, in my view I think that's that's probably going to be the most important benchmark and I don't I don't think so anyone has actually made this yet.

s898
01:22:28.159 --> 01:22:34.719
Um I don't know maybe someone in the audience or you know you guys can go as teams I don't know make a startup or something you know do that.

s899
01:22:34.719 --> 01:22:37.360
Um and I feel like that benchmark will be very very important.

s900
01:22:37.360 --> 01:22:40.239
Um yeah

s901
01:22:45.360 --> 01:22:49.360
benchmarks we can trust today none of them.

s902
01:22:49.360 --> 01:22:50.880
take an average of all of them.

s903
01:22:50.880 --> 01:22:55.360
To be honest, probably the best approach is just vibe uh vibe checking.

s904
01:22:55.360 --> 01:22:57.600
Try all of them and see which one you like the best.

s905
01:22:57.600 --> 01:23:11.760
Um to be completely honest, I just you know like these benchmarks h like the main the main issue I have with benchmarks is for example um you know I mean like this one right

s906
01:23:11.760 --> 01:23:16.159
this one I mean even every single day the benchmark can change.

s907
01:23:16.159 --> 01:23:18.080
So we can't trust the benchmarks anymore.

s908
01:23:18.080 --> 01:23:22.080
So my fundamental view is do not trust any benchmarks.

s909
01:23:22.080 --> 01:23:25.600
Take an average and then okay then main question is who's taking the average?

s910
01:23:25.600 --> 01:23:29.040
I guess artificial artificial analysis has some average.

s911
01:23:29.040 --> 01:23:31.199
The only problem is they have some weightings for the weight.

s912
01:23:31.199 --> 01:23:34.159
Um you know each benchmark has a weight.

s913
01:23:34.159 --> 01:23:37.040
So now the question is you know what is the waiting of each benchmark.

s914
01:23:37.040 --> 01:23:39.199
You know you can't just take like a dumb average.

s915
01:23:39.199 --> 01:23:44.239
Um you know you can't just say you know 10 benchmarks divided by 10.

s916
01:23:44.239 --> 01:23:45.600
Um that's probably not going to work.

s917
01:23:45.600 --> 01:23:47.600
So the main question is how do you even do the waiting?

s918
01:23:47.600 --> 01:23:48.639
That's another problem.

s919
01:23:48.639 --> 01:23:53.520
Um, so I think in general it's based on vibe checking, I guess.

s920
01:23:53.520 --> 01:23:56.159
Yeah, I guess I don't have an answer for that.

s921
01:23:56.159 --> 01:23:58.880
Um, any other questions?

s922
01:23:58.880 --> 01:24:00.880
Yes.

s923
01:24:43.920 --> 01:24:45.920
question.

s924
01:24:54.719 --> 01:24:55.600
Yes.

s925
01:24:55.600 --> 01:25:02.159
So that way Good matter.

s926
01:25:03.679 --> 01:25:04.320
You're correct.

s927
01:25:04.320 --> 01:25:09.760
So the question was in terms of because we bench pro for example you call a model.

s928
01:25:09.760 --> 01:25:11.199
The question is what model?

s929
01:25:11.199 --> 01:25:12.400
Could it be 4.8?

s930
01:25:12.400 --> 01:25:17.920
Could it be GB 5.5 and you call this model to verify the benchmark?

s931
01:25:17.920 --> 01:25:21.920
Um you and so the question was can you use an open source model instead?

s932
01:25:21.920 --> 01:25:24.480
So then now you you have a controlled environment.

s933
01:25:24.480 --> 01:25:25.760
So yes you can.

s934
01:25:25.760 --> 01:25:38.639
But remember there is a problem because even open source models itself have bugs times you know the inference engines have bugs times the inference providers have bugs and accuracy degradation.

s935
01:25:38.639 --> 01:25:40.000
So it's you're correct.

s936
01:25:40.000 --> 01:25:59.600
Um so the main question is we need to have some one or some organization you know some person or some whatever committee that we can investigate you know which engine did you use do not update the engine you know the engine must be the same you know the weights must have not changed so there's many many many

s937
01:25:59.600 --> 01:26:09.600
problems with this approach um but I do agree as open you can use an open source model but it's not it doesn't solve the other problems um yeah does that

s938
01:26:09.600 --> 01:26:10.719
Okay.

s939
01:26:10.719 --> 01:26:14.480
Um, so the next section I'm going to be talking about is cyber security and regulation.

s940
01:26:14.480 --> 01:26:17.199
Um, this is a interesting topic.

s941
01:26:17.199 --> 01:26:20.159
Um, so I'm not sure if you have all folks have seen this plot.

s942
01:26:20.159 --> 01:26:22.639
It shows the AI security institutes.

s943
01:26:22.639 --> 01:26:24.400
Um, I think this is from the UK.

s944
01:26:24.400 --> 01:26:28.880
Um, they show the performance of models based on some cyber security task.

s945
01:26:28.880 --> 01:26:33.440
Um, and they show that mythos preview seems to be the best.

s946
01:26:33.440 --> 01:26:37.280
Um, you know, with GBD 5.5 cybar, you know, preview and so on.

s947
01:26:37.280 --> 01:26:39.120
Um, they show this benchmark.

s948
01:26:39.120 --> 01:26:51.120
Uh and again previously as I mentioned weird ML is a better ben in my view okay this is just my take weird ML is a better benchmark in general

s949
01:26:51.120 --> 01:27:04.159
for benchmarking intelligence on models and the reason why is because it doesn't actually it doesn't actually follow the trend of reasoning versus non-reasoning remembering reasoning previously I think I okay I don't have it um reasoning

s950
01:27:04.159 --> 01:27:13.199
um the reasoning models doubling time reduced by half to 3.5 months So remember, you just need to wait 3.5 months and the models capabilities will double.

s951
01:27:13.199 --> 01:27:16.159
Um and the non-reasoning was 7 months.

s952
01:27:16.159 --> 01:27:20.000
Um so you need to wait seven months for the models to double in capability.

s953
01:27:20.000 --> 01:27:22.800
Um but weird ML did not actually have this trend.

s954
01:27:22.800 --> 01:27:27.360
Um the weird ML benchmark showed that actually the trend was like there is no trend.

s955
01:27:27.360 --> 01:27:38.880
Um, and I think I'll just talk about this, you know, like one of the biggest problems of benchmarks is you need to constantly reinvent yourself and do reweings of

s956
01:27:38.880 --> 01:27:40.320
combinations of benchmarks.

s957
01:27:40.320 --> 01:27:48.239
For example, artificial analysis just recently released, you know, their new v4.1 benchmark and they showed the waiting of the benchmarks.

s958
01:27:48.239 --> 01:27:52.560
Um, you know, GDP vow is 20%, terminal bench is 16% and so on.

s959
01:27:52.560 --> 01:27:59.040
Um and so they they designed these numbers as waitings for each of those benchmarks and then they've averaged it up together.

s960
01:27:59.040 --> 01:28:02.639
Um so the main question is how do you actually determine these numbers?

s961
01:28:02.639 --> 01:28:04.800
Um and so this is more like a human approach.

s962
01:28:04.800 --> 01:28:06.719
You know you have to determine these numbers.

s963
01:28:06.719 --> 01:28:22.560
um you know arc AGI kind of saturated on ARC AGI 1 and so that's why we have ARC AGI 2 and that is also why we have ARC AGI 3 and my you know I guess once ARC AGI 3 is saturated then we have ARC AGI 4 5 6 7 whatever

s964
01:28:22.560 --> 01:28:34.320
um and the main point is once you have benchmarks is it called good art I don't remember um the good the benchmark itself becomes useless because you know models will start benchmarking on this

s965
01:28:34.800 --> 01:28:46.239
so one of the biggest problems of these larger models for cyber security for example um is mythos actually dramatically went out of the trend um and that is why you know

s966
01:28:46.239 --> 01:28:58.960
many people are afraid of these you know mythos you know GPD 5 point 5.6 you they're afraid of these models because it went out of trend um you can see that mythos dramatically went out of trend um

s967
01:28:58.960 --> 01:29:04.480
and you know even you know you know GP 5.6 didn't really release that many benchmarks because it was in preview mode.

s968
01:29:04.480 --> 01:29:06.400
Um, so this is from their system card.

s969
01:29:06.400 --> 01:29:10.719
They showed for cyber security that Guby 5.6 does very very well.

s970
01:29:10.719 --> 01:29:18.480
Um, in fact because Guby because Guby 5.6 they I think they only did terminal bench as their benchmark.

s971
01:29:18.480 --> 01:29:20.400
They did not benchmark on anything else.

s972
01:29:20.400 --> 01:29:25.199
Um, they did have in their system card they did have one benchmark which is very important.

s973
01:29:25.199 --> 01:29:28.880
Um, and this is called the internal research debugging evaluation.

s974
01:29:28.880 --> 01:29:33.440
Um, and this is OpenAI's own set of set of questions.

s975
01:29:33.440 --> 01:29:41.600
So, you know, the custom open source, you know, if you want to, it's their own set of 10 questions or whatever that they benchmarked GBD 5.6 on.

s976
01:29:41.600 --> 01:29:45.040
Um, and according to them, it does very very it does better.

s977
01:29:45.040 --> 01:29:46.719
Okay, I was going to say very, very well, but it's not.

s978
01:29:46.719 --> 01:29:48.000
Um, it does better.

s979
01:29:48.000 --> 01:29:51.440
Um, and you can see that GBD, it's actually kind of interesting.

s980
01:29:51.440 --> 01:30:08.239
GBD 5.5 did worse than GBD 5.5 a four um for OpenAI's own internal um research evaluation um and you know GBD 5.6 definitely does much better right you can see that the G GBD 5.6 soul, you know, if you extend it, it does much better.

s981
01:30:08.239 --> 01:30:13.600
Um, but interestingly, Terara does better um somewhat sometimes.

s982
01:30:13.600 --> 01:30:16.400
Um, yeah.

s983
01:30:17.920 --> 01:30:26.239
And, you know, one of the biggest problems of these models that are getting getting better and better and better is I don't know if you guys know that, you know, open-source

s984
01:30:26.239 --> 01:30:28.639
exploits are getting worse and worse and worse.

s985
01:30:28.639 --> 01:30:42.080
Um and so the high exploit ratio you know number of critical vulnerabilities that were discovered has skyrocketed you know recently you know every single week or day some sort of open source package gets compromised

s986
01:30:42.080 --> 01:30:56.400
um and they actually you know this plot shows that it's getting very problematic um and so you know claude mythos was released at this dotted line where you know most people they're not sure if it's because of claude mythos that these vulnerabilities

s987
01:30:56.400 --> 01:31:07.920
are increasing most likely it's because open source you know we use lots of models call them many many many many times and we can you know automatically find exploits in these models

s988
01:31:08.480 --> 01:31:20.639
but you know there is actually another point so in hacken news someone posted about this um is it just mythos and juby 5.6 six that do good on finding cyber security issues.

s989
01:31:20.639 --> 01:31:24.239
It's not actually open source models also do very well.

s990
01:31:24.239 --> 01:31:30.719
Um open source models do extremely well in finding cyber security threats and issues.

s991
01:31:30.719 --> 01:31:34.480
Um you know there is some discussion on hacker news you know is this actually true or false?

s992
01:31:34.480 --> 01:31:48.400
Um but you know according to some you know some researchers and cyber security people the main reason why you know mythos looked like it was very good on cyber security is because they bothered to actually check the open source code.

s993
01:31:48.400 --> 01:31:58.719
Um and so if you actually give the open source models the full code base of these open source libraries they will find the bugs you know they will find cyber security issues.

s994
01:31:58.719 --> 01:32:01.600
Um and all you need to do is core the model.

s995
01:32:01.600 --> 01:32:12.239
Um and so I feel like you know that's the fundamental problem is mythos seems very powerful not because the model is powerful but because they actually bothered to test on all open source repos.

s996
01:32:12.239 --> 01:32:25.600
Um and so if you do you know if you call all these open source models to detect for bugs for cyber security issues you will find bugs and you know as as you know recently

s997
01:32:25.600 --> 01:32:30.719
you know as everyone knows fable is still banned for the majority of everyone.

s998
01:32:30.719 --> 01:32:34.719
Um, and GBD 5.6, you know, is delayed a staggered release, right?

s999
01:32:34.719 --> 01:32:37.199
So like Guby 5.6 preview was on Friday, right?

s1000
01:32:37.199 --> 01:32:40.800
So like a few days ago, and they said they're not going to be releasing to everyone.

s1001
01:32:40.800 --> 01:32:51.600
Um, and the main questions are, you know, in the open source world, in the clos world, people are asking, do we need a license to use these AI models

s1002
01:32:51.600 --> 01:32:52.800
for everyone?

s1003
01:32:52.800 --> 01:32:58.080
You know, like everyone in this room now, we have to have a license to use the models, like a driver's license.

s1004
01:32:58.080 --> 01:32:59.679
Um, do we need to get that?

s1005
01:32:59.679 --> 01:33:03.040
um is there going to be a delay in all of these releases?

s1006
01:33:03.040 --> 01:33:08.560
So every single time when a new model gets released only the trusted providers get these models.

s1007
01:33:08.560 --> 01:33:21.760
Um the next most important question how about open-source models you know okay the government the US government currently is like you know trying to like control Fable GPD 5.6

s1008
01:33:21.760 --> 01:33:25.199
The main question now is what do we do about open source models?

s1009
01:33:25.199 --> 01:33:27.840
You know, open models, open weight models.

s1010
01:33:27.840 --> 01:33:32.159
What will the government do to control the open source space?

s1011
01:33:32.159 --> 01:33:39.120
To be completely honest, I was quite surprised the government acted this early um in doing GBD 5.6 and fable control, right?

s1012
01:33:39.120 --> 01:33:42.719
I thought it was like maybe the end of the year or next year, but it seems like it's now.

s1013
01:33:42.719 --> 01:33:46.000
Um so the next question is what will happen to open source models?

s1014
01:33:46.000 --> 01:33:49.040
Will the government start controlling open source models?

s1015
01:33:49.040 --> 01:34:00.159
Um and the fundamental question is what defines frontier intelligence like the reason why the government is you know they're controlling these models is because they're very very powerful.

s1016
01:34:00.159 --> 01:34:03.679
Um so the main question is what actually defines intelligence?

s1017
01:34:03.679 --> 01:34:06.639
You know which benchmark do we use?

s1018
01:34:06.639 --> 01:34:12.480
Is it just based on one trillion parameters like you know how do we define whether a model can be banned or unbanned?

s1019
01:34:12.480 --> 01:34:14.639
Um and that is a very very important question.

s1020
01:34:14.639 --> 01:34:21.840
Um and we will we have a dark web of open models now you know do we need to torrent open models?

s1021
01:34:21.840 --> 01:34:32.480
Um and the most important question what is inference what are inference providers going to do now you know assuming assuming that the government has some sort of regulation on even open models.

s1022
01:34:32.480 --> 01:34:35.600
Um what is the inference prov what are they going to do?

s1023
01:34:35.600 --> 01:34:37.520
You know what are the inference providers going to do?

s1024
01:34:37.520 --> 01:34:39.040
Do they need to have license?

s1025
01:34:39.040 --> 01:34:43.760
Do they need to check that everyone has a license before you can use the model?

s1026
01:34:43.760 --> 01:34:44.880
Um or something like that.

s1027
01:34:44.880 --> 01:34:56.880
Um and so like you know these are very important questions that you know the government is currently like you know and the industry you know the entire AI ecosystem and industry we are trying to like you know what are the answers to these questions.

s1028
01:34:56.880 --> 01:35:04.800
Um and obviously you know if you were the government if I was the government it makes sense you know they do not want their critical infrastructure to be hacked.

s1029
01:35:04.800 --> 01:35:08.639
You know, remember open- source exploits are skyrocketing.

s1030
01:35:08.639 --> 01:35:15.520
If you change that y-axis, you know, not open source exploits, but like critical infrastructure exploits, you know, obviously the government's scared.

s1031
01:35:15.520 --> 01:35:18.320
Um, so it makes sense for them to like stagger the release.

s1032
01:35:18.320 --> 01:35:25.040
Um, but the main question is, you know, we're still in this we're still in this fog of war type approach, you know.

s1033
01:35:25.040 --> 01:35:30.159
Okay, not fog of war, just fog, a foggy, you know, we don't know what will happen for regulation.

s1034
01:35:30.159 --> 01:35:32.960
Um, yeah, that's very problematic.

s1035
01:35:32.960 --> 01:35:34.560
Um yeah.

s1036
01:35:34.560 --> 01:35:35.360
Oh okay.

s1037
01:35:35.360 --> 01:35:41.040
Anyone have any questions for cyber security regulation policy or whatever?

s1038
01:35:41.040 --> 01:35:45.840
Um or any takes as well questions?

s1039
01:35:45.840 --> 01:35:47.840
Yes.

s1040
01:36:02.080 --> 01:36:03.199
That is a good question.

s1041
01:36:03.199 --> 01:36:14.800
So is it is open source so the scare of open source models is it because you know anthropic keeps screaming about open source is bad open source is bad you know every single day open source is bad

s1042
01:36:14.800 --> 01:36:25.840
um yes and no I feel like it's true that you know there are some players in the closed source industry they want to shut down the open source ecosystem

s1043
01:36:25.840 --> 01:36:34.320
their view is if you give open source to anyone they will start hacking you know critical infrastructure they will start doing bad behavior.

s1044
01:36:34.320 --> 01:36:36.320
Um, and so that's kind of their view.

s1045
01:36:36.320 --> 01:36:42.639
Um, so yes, I agree that some of the closed source labs have caused this problem.

s1046
01:36:42.639 --> 01:36:50.800
Um, but it's actually kind of funny because currently the government is regulating them first and open source is still a question mark.

s1047
01:36:50.800 --> 01:36:54.320
Um, and so like it's kind of like I don't know, they probably stabbed themselves in the foot or something.

s1048
01:36:54.320 --> 01:36:56.320
I don't know whatever whatever the phrase is.

s1049
01:36:56.320 --> 01:37:05.840
Um but I feel like it's they did cause some controversy in terms of like saying open source is bad but in general open source models are actually good.

s1050
01:37:05.840 --> 01:37:18.960
Um so you could I mean theoretically you can use an open source model and you know run this on all repos and you will be able to find exploits

s1051
01:37:18.960 --> 01:37:20.320
and you can exploit.

s1052
01:37:20.320 --> 01:37:21.600
So they're not wrong.

s1053
01:37:21.600 --> 01:37:25.600
Um but I feel like you know who has the infrastructure to do this?

s1054
01:37:25.600 --> 01:37:28.480
um you know, GitHub might automatically detect you and ban you or something.

s1055
01:37:28.480 --> 01:37:29.040
I don't know.

s1056
01:37:29.040 --> 01:37:34.960
There's many there's many um layers of security for each section.

s1057
01:37:34.960 --> 01:37:36.639
Um and so like I I don't know.

s1058
01:37:36.639 --> 01:37:41.760
I I feel like it's somewhat overblown, but it is it is it's not 0% probability.

s1059
01:37:41.760 --> 01:37:42.800
So it is a problem.

s1060
01:37:42.800 --> 01:37:47.600
Um yeah, if that answers your question, but okay.

s1061
01:37:47.600 --> 01:37:48.080
Yes.

s1062
01:37:48.080 --> 01:37:50.239
So now we're going to be talking about kernels.

s1063
01:37:50.239 --> 01:37:55.600
Um so previously you know this is my favorite plot as usual.

s1064
01:37:55.600 --> 01:38:05.679
Um you know if we were in a different future you know if we were in a different timeline that we did not discover 01 preview models would have plateaued.

s1065
01:38:05.679 --> 01:38:07.520
I think that's the fundamental point of this plot.

s1066
01:38:07.520 --> 01:38:15.040
It shows that if we have never discovered reasoning we have never discovered 01 whatever we will have plateaued.

s1067
01:38:15.040 --> 01:38:17.119
We will have plateaued in terms of accuracy.

s1068
01:38:17.119 --> 01:38:18.400
Um and that is not good.

s1069
01:38:18.400 --> 01:38:25.119
Um and because we have discovered this new paradigm of scaling, you know, models have continuously scaled even further.

s1070
01:38:25.119 --> 01:38:37.679
Um but my take is the reason why we have stopped scaling um based on you know the old approach is because the old approach only focused on hardware optimizations.

s1071
01:38:37.679 --> 01:38:42.159
We now have to move over to software optimizations and algorithmic optimizations.

s1072
01:38:42.159 --> 01:38:47.679
Um we you know we need to have new inventions of how do we scale AI even further.

s1073
01:38:47.679 --> 01:38:52.560
Um and we can't just rely on doing 10 trillion parameters or you know making the model bigger and bigger and bigger bigger.

s1074
01:38:52.560 --> 01:38:56.320
Um for example you know we have to do float a reinforcement learning.

s1075
01:38:56.320 --> 01:39:03.199
Um so if pytorch has this methodology where you can do floatate float for different precisions to make training faster.

s1076
01:39:03.199 --> 01:39:04.639
Um and that is one way.

s1077
01:39:04.639 --> 01:39:11.119
Um another way for example as a software approach for example as I previously said we found some bit you know issues in gradient accumulation.

s1078
01:39:11.119 --> 01:39:18.400
Um so when you do gradient accumulation um it was actually it was not calculated correctly during the loss calculation.

s1079
01:39:18.400 --> 01:39:24.320
Um and you can actually increase accuracy by 1 to 3% if you fix this small little issue.

s1080
01:39:24.880 --> 01:39:25.040
Yeah.

s1081
01:39:25.040 --> 01:39:29.199
So like you know the universal gradient accumulation bug fix was a software fix.

s1082
01:39:29.199 --> 01:39:30.960
It is not a hardware fix.

s1083
01:39:30.960 --> 01:39:36.080
Um and so the fundamental view is you need to do more and more software changes.

s1084
01:39:36.080 --> 01:39:36.239
Right?

s1085
01:39:36.239 --> 01:39:47.440
Another one for example Snowflake we collaborated them to make context long context fine-tuning 500k contact length this was all software improvements um another one is you know 12 times faster MLB training

s1086
01:39:47.440 --> 01:39:57.920
this is another software improvement um deepseeek you know they released something called deep spark which was just a few days um and they showed that they can make inference

s1087
01:39:57.920 --> 01:40:07.199
you know 50 50 to 600% faster so six times faster than just normal MTP um and so this is a software ware methodology, right?

s1088
01:40:07.199 --> 01:40:09.920
Not a hardware methodology.

s1089
01:40:09.920 --> 01:40:12.080
And you know, diffusion Gemma, right?

s1090
01:40:12.080 --> 01:40:18.880
Gemma released a new diffusion model showcasing that you can get 2,000 tokens per second by using a new architecture, right?

s1091
01:40:18.880 --> 01:40:25.920
So using diffusion LLMs to do faster inference and again this is a software change.

s1092
01:40:25.920 --> 01:40:32.880
And my main point is is that in general, hardware innovations are getting less and less important.

s1093
01:40:32.880 --> 01:40:35.920
Um and hardware innovations are actually slowing down.

s1094
01:40:35.920 --> 01:40:48.320
Um so it's actually kind of interesting intelligence you know the scaling of you know intelligence in general it's kind of like Mo's law um it's kind of like a there is a relentless

s1095
01:40:48.320 --> 01:40:58.480
progress relentless approach to increase intelligence and the same with Mo's law um and so like in general you can see that you this is Mo's law over here the number of transistors

s1096
01:40:58.480 --> 01:41:09.600
has continuously increased um but you know single performance is not increasing it has staggered Um and so this is kind of like you know this kind of reminds me of

s1097
01:41:09.600 --> 01:41:26.320
you know this plot right scaling intelligence in terms of parameters probably has plateaued most likely you know hardware performance pre-training whatever we now need to go into this new reasoning paradigm to scale even further um so it kind of is like similar to the moors law

s1098
01:41:26.320 --> 01:41:36.800
type graph um kind of um and you can see if you see on this side the number representation of GP so my GPU is getting faster and faster and faster, right?

s1099
01:41:36.800 --> 01:41:39.920
It's not actually the GPU itself that's getting faster and faster and faster.

s1100
01:41:39.920 --> 01:41:43.280
Um, it's the represent number representation, right?

s1101
01:41:43.280 --> 01:41:50.000
So like they changed from float 32 all the way to float 4. Um, and this made GPUs 32 times faster.

s1102
01:41:50.000 --> 01:41:52.719
Um, so it's not eight times faster, right?

s1103
01:41:52.719 --> 01:41:55.440
It's not 32 divided by 4 is eight times faster.

s1104
01:41:55.440 --> 01:41:57.520
It's 32 times faster.

s1105
01:41:57.520 --> 01:42:02.719
And the reason why is because of tensores, um, you know, the smaller mantesses, um, and so on.

s1106
01:42:02.719 --> 01:42:12.080
Um, and so like you can actually see, you know, even tensors with the introduction of tensores, it made the GPUs 12 times faster and so on.

s1107
01:42:12.080 --> 01:42:16.000
Actually, if you made the GPU smaller and smaller and smaller, it only made it three times faster.

s1108
01:42:16.000 --> 01:42:17.679
It's not even that important anymore.

s1109
01:42:17.679 --> 01:42:27.040
Um, and if you look at this plot, we are now at float 4. So most of the GPUs that we have now are at float four.

s1110
01:42:27.040 --> 01:42:28.320
What is next?

s1111
01:42:28.320 --> 01:42:32.719
Are we going to be having float three, float two, float one?

s1112
01:42:32.719 --> 01:42:34.239
Are we going to have float zero?

s1113
01:42:34.239 --> 01:42:35.119
Okay, no such thing.

s1114
01:42:35.119 --> 01:42:39.199
But anyways, the point is hardware is kind of at its limits, right?

s1115
01:42:39.199 --> 01:42:42.560
We're already at float 4. What is next?

s1116
01:42:42.560 --> 01:42:43.679
There is nothing next.

s1117
01:42:43.679 --> 01:42:47.119
Um, and so the so the answer to this question is there is nothing next.

s1118
01:42:47.119 --> 01:42:49.600
Um, and so now we need to move over to software, right?

s1119
01:42:49.600 --> 01:42:51.199
How do we make new algorithms?

s1120
01:42:51.199 --> 01:42:54.080
How do we make new methodologies to continue scaling?

s1121
01:42:54.080 --> 01:42:56.560
Um, I also made this table, right?

s1122
01:42:56.560 --> 01:43:14.880
I previously said why is you know you use float 32 um we change it to float 4 why is it why is it not eight times faster um and instead it's 32 times faster right why is it 32 times faster and the reason is because when you use when you do floatingoint precision you have an exponent and

s1123
01:43:14.880 --> 01:43:28.880
a menta um and the transistor space the transistor space is the exponent plus the menta squared um and so the trick is if you make them in Tesla smaller and smaller and smaller,

s1124
01:43:28.880 --> 01:43:31.840
you square their number of improvements, right?

s1125
01:43:31.840 --> 01:43:37.119
So, float 32, float 32, you needed 537 transistors around, right?

s1126
01:43:37.119 --> 01:43:38.800
537 transistors.

s1127
01:43:38.800 --> 01:43:44.639
Um, to go from float 32 to float 16, you only need 105 transistors.

s1128
01:43:44.639 --> 01:43:49.520
So, actually you made you made the number of transistors five times more, right?

s1129
01:43:49.520 --> 01:43:52.239
So, not two times, it's five times.

s1130
01:43:52.239 --> 01:43:54.480
Um, and so on so on so on.

s1131
01:43:54.480 --> 01:43:59.520
Um, so you know, I guess you can go to 1.58 bit.

s1132
01:43:59.520 --> 01:44:00.719
I guess you can do that.

s1133
01:44:00.719 --> 01:44:05.280
Um, but it's actually kind of interesting because 1.58 bit um is actually not that much faster.

s1134
01:44:05.280 --> 01:44:21.119
Um, so 1.58 bit is actually not that much faster um than float 8. Um, if you use um, you know, seven seven exponent and Messa 2. Um, there is another 1.58 bit which you use float 4. Um, so float 4 is 179 times faster than float 32.

s1135
01:44:21.119 --> 01:44:38.239
Um and the main question is we are already at three transistors right we are already around three transistors what are we going to do next two transistors or like one transistor so don't like you know most likely GPUs are not going to be getting faster um that's the fundamental question of this plot so GPUs are not going

s1136
01:44:38.239 --> 01:44:52.400
to be getting faster instead we need to focus on kernels right how do we make better kernels better algorithms how do we scale this instead right don't do don't do hardware optimizations

s1137
01:44:52.400 --> 01:44:53.199
anymore.

s1138
01:44:53.199 --> 01:44:57.199
Instead, how do we do, you know, these optimizations?

s1139
01:44:57.199 --> 01:45:02.719
Um, and so one of my favorite tools to use, you know, everyone should use this is just use torch compile.

s1140
01:45:02.719 --> 01:45:13.119
Um, so in my, you know, it's the modern, you know, the modern time, do not, as advice, do not learn how to write custom kernels.

s1141
01:45:13.119 --> 01:45:14.239
That is advice.

s1142
01:45:14.239 --> 01:45:16.400
Do not do kernel writing.

s1143
01:45:16.400 --> 01:45:21.360
Um and the reason why is because torch compile will take over all of kernel writing.

s1144
01:45:21.360 --> 01:45:33.520
Um so you can see for example this plot um torch compile was a red line right performance it doesn't look like it's doing very well right it does not look like it's doing very well versus handwritten

s1145
01:45:33.520 --> 01:45:46.480
kernels right handwritten kernels are the other ones right so torch compile doesn't look like it's doing very well but that's because that's an old PyTorch version if you have a newer PyTorch version torch compile wins

s1146
01:45:46.480 --> 01:45:52.239
dramatically right that's the orange line um and all of these are handwritten kernels.

s1147
01:45:52.239 --> 01:45:55.520
Oh, the okay the black line is torch compile plus no fusion.

s1148
01:45:55.520 --> 01:45:57.440
Um so that's another torch compile method.

s1149
01:45:57.440 --> 01:46:05.040
Um but the red line, the green line and the blue line okay the the blue line is just no torch compile just normal PyTorch.

s1150
01:46:05.040 --> 01:46:14.800
Um but the green line and the black the green line and the red line are handwritten kernels and you can see it does even worse than torch compile.

s1151
01:46:14.800 --> 01:46:17.280
So like my viewers like what's the point of writing kernels?

s1152
01:46:17.280 --> 01:46:19.040
Torch compile does even better than you.

s1153
01:46:19.040 --> 01:46:35.360
Um so the main point is you should always firstly look at torch compile right before you write a kernel use torch compile first do not start learning how to do triton or you know cuda or whatever is your favorite coding language for kernels

s1154
01:46:35.360 --> 01:46:49.600
don't do that instead use torch compile um even worse like you know this this was RMS norm um you know this is layer norm torch compile wins dramatically um you know versus handwritten kernels.

s1155
01:46:49.600 --> 01:46:54.400
So I would not you know definitely only use torch compile as your first try.

s1156
01:46:54.400 --> 01:46:57.360
Um do not write kernels first.

s1157
01:46:57.360 --> 01:46:59.920
Use torch compile.

s1158
01:47:00.400 --> 01:47:06.639
So the main takeaway is algorithms are much more important than hardware or whatever handwritten kernels.

s1159
01:47:06.639 --> 01:47:06.880
Right?

s1160
01:47:06.880 --> 01:47:19.600
Remember deepseek released deep you know deep you know deep spark you know there's other algorithms for speculative decoding like MTP D flash DSpark whatever all of these are algorithmic

s1161
01:47:19.600 --> 01:47:30.960
improvements and these made inference two times to six times faster right it wasn't like new some new hardware it wasn't some new hardware which made inference faster it was algorithms

s1162
01:47:30.960 --> 01:47:40.719
which made inference faster right flash attention flash you know FA2 FA3 three, flash attention four, flash attention five, six, seven, whatever, right?

s1163
01:47:40.719 --> 01:47:44.000
All of these are algorithmic improvements, right?

s1164
01:47:44.000 --> 01:47:48.800
Flash attention was essentially a trick to do memory movement much better.

s1165
01:47:48.800 --> 01:47:54.320
Um, so how do we like orchestrate memory movement and use the caching structure of the GPUs much better?

s1166
01:47:54.320 --> 01:47:57.920
Um, and so flash attention is also a algorithm.

s1167
01:47:57.920 --> 01:48:04.000
Gradient checkpointing, you know, one of the most important algorithms for training is gradient checkpointing.

s1168
01:48:04.000 --> 01:48:07.600
Um, and all it does is you do not save all the activations.

s1169
01:48:07.600 --> 01:48:11.679
You do a trick where you only save the activations for every single layer.

s1170
01:48:11.679 --> 01:48:15.119
Um, and then you skip all the intermediate activations in each layer.

s1171
01:48:15.119 --> 01:48:18.000
Um, and then you recomp compute the activations.

s1172
01:48:18.000 --> 01:48:22.719
Um, and gradient checkpointing saves memory by dramatic amounts by like 70%.

s1173
01:48:22.719 --> 01:48:27.360
Um, 70% memory reduction with no change in accuracy.

s1174
01:48:27.360 --> 01:48:31.520
And okay, training is a little bit slower maybe by 10% to 15%.

s1175
01:48:31.520 --> 01:48:35.679
Um and you know grading check checkpointing was an algorithm.

s1176
01:48:35.679 --> 01:48:46.080
Um and you know like in general you should also try to understand you know what is the new data processing tricks you know how do we like you know stagger data you know do we do

s1177
01:48:46.080 --> 01:48:55.520
do we do curriculum learning or something like that I don't know um what you know how do we clean the data set before we actually pre-train the model there are many tricks you can employ for data processing

s1178
01:48:55.520 --> 01:49:06.560
um and obviously you know there is still a group of people I don't know I take an opposite view there is a group of people who think mega kernels are the latest and greatest for kernels.

s1179
01:49:06.560 --> 01:49:07.920
You know what is a mega kernel?

s1180
01:49:07.920 --> 01:49:15.920
A mega kernel is when you take an entire you take an entire implementation of a model and it's just one kernel like one large kernel.

s1181
01:49:15.920 --> 01:49:28.719
Um ah maybe it's useful who knows um you know you know Nvidia you know Nvidia has a acquired Grock or something um and you know their view is for example you have two different

s1182
01:49:28.719 --> 01:49:39.840
systems right the LPU which is the Grock system does the decoding right so like the MLP layers the layers does the decoding and then the GPU so the Nvidia GPUs

s1183
01:49:39.840 --> 01:49:53.679
does the attention and the prefield um and so in general you know we might even have a future We have different types of hardware systems you know we have asex which are you know specially designed chips for you know computation

s1184
01:49:53.679 --> 01:49:56.000
and we have generalized systems like GPUs.

s1185
01:49:56.000 --> 01:49:59.600
Um and these asex and GPUs will collaborate with each other.

s1186
01:49:59.600 --> 01:50:11.280
Um so for example the attention you know the the attention will be for the GPUs and they will transfer over to the LPU to do you know the MLPS

s1187
01:50:11.280 --> 01:50:11.840
and so on.

s1188
01:50:11.840 --> 01:50:21.440
Um and then this is like a dance you know between them and you can also do like pipelining right you can imagine that there's like many many many replicas of this and they can like you know serve

s1189
01:50:21.440 --> 01:50:24.800
you know 20 people or you know 1,000 people in one go.

s1190
01:50:24.800 --> 01:50:32.880
Um and yeah so this is like another approach and you know this in my view this is kind of an opposite approach of mega kernels.

s1191
01:50:32.880 --> 01:50:43.840
So as a mega kernel your view is you want to combine the goal is to make the goal of a mega kernel is to make one kernel for the full forward path of a language model.

s1192
01:50:43.840 --> 01:50:56.000
Um and once you make one once you once you are able to make the language model the forward path into one kernel you can now make the entire language model with 32 layers as one kernel.

s1193
01:50:56.000 --> 01:50:56.239
Right?

s1194
01:50:56.239 --> 01:51:02.000
You can extend this and because the whole language model is one kernel you can even further extend it.

s1195
01:51:02.000 --> 01:51:02.400
Right?

s1196
01:51:02.400 --> 01:51:07.520
the prediction of the second token, the third token, the fourth token, the sixth token can all be just one kernel.

s1197
01:51:07.520 --> 01:51:11.199
Um, and unfortunately, this is very hard to do.

s1198
01:51:11.199 --> 01:51:14.719
It's very hard because attention is the problem, right?

s1199
01:51:14.719 --> 01:51:18.239
Attention has to see the tokens in the future.

s1200
01:51:18.239 --> 01:51:20.320
See the tokens in the past, not the future.

s1201
01:51:20.320 --> 01:51:20.960
That's cheating.

s1202
01:51:20.960 --> 01:51:23.119
Um, you have to see the tokens in the past.

s1203
01:51:23.119 --> 01:51:24.480
And that is a fundamental problem.

s1204
01:51:24.480 --> 01:51:32.639
Um and it's very hard to you know it's very hard to make a mega kernel to combine attention and the MLE or MLP layers.

s1205
01:51:32.639 --> 01:51:34.000
It's extremely complicated.

s1206
01:51:34.000 --> 01:51:42.239
So in general what what people do is they will make two kernels right one kernel for the attention part and the other kernel for the rest.

s1207
01:51:42.239 --> 01:51:45.280
Um and so you will see there are two kernels.

s1208
01:51:45.280 --> 01:51:51.760
Um and yeah so it's very hard to make one mega kernel but you can make two kernels.

s1209
01:51:52.719 --> 01:51:53.040
Yes.

s1210
01:51:53.040 --> 01:51:53.199
Okay.

s1211
01:51:53.199 --> 01:51:54.159
Any other questions?

s1212
01:51:54.159 --> 01:51:55.760
Any questions for kernels?

s1213
01:51:55.760 --> 01:51:57.760
So the main takeaway for Yes, a question.

s1214
01:51:57.760 --> 01:51:59.520
Yes, that is a very good question.

s1215
01:51:59.520 --> 01:52:04.080
So the question was because there's so many knobs for torch compile like 1,00 or something.

s1216
01:52:04.080 --> 01:52:08.320
How do we reduce the experimentation time to like you know find which knob is the best?

s1217
01:52:08.320 --> 01:52:14.000
Um so luckily we have something called bisection or binary search.

s1218
01:52:14.000 --> 01:52:14.800
That's the trick.

s1219
01:52:14.800 --> 01:52:19.520
So what we'll do is instead of checking every single 10,00 combination randomly sample.

s1220
01:52:19.520 --> 01:52:22.080
So random you do randomize bisection.

s1221
01:52:22.080 --> 01:52:25.920
You randomly sample 50% of the, you know, flags.

s1222
01:52:25.920 --> 01:52:31.119
You turn it on versus turning it off and then benchmark which one is better.

s1223
01:52:31.119 --> 01:52:34.880
And whichever one is better, you then narrow down the search.

s1224
01:52:34.880 --> 01:52:38.639
You again do 50% and 50% and 50% and 50%.

s1225
01:52:38.639 --> 01:52:40.880
So it's actually log two of 10,00.

s1226
01:52:40.880 --> 01:52:42.560
I don't know what that what is log 2 of 10,000.

s1227
01:52:42.560 --> 01:52:43.520
I don't know what that is.

s1228
01:52:43.520 --> 01:52:47.360
Um two times I don't know.

s1229
01:52:47.360 --> 01:52:48.800
Anyways, log 200.

s1230
01:52:48.800 --> 01:52:50.800
I think you need to do 30 steps, I think.

s1231
01:52:50.800 --> 01:52:51.199
I don't know.

s1232
01:52:51.199 --> 01:52:55.360
I don't remember whatever 2 to the^ of something is equal to 1,00 then log it.

s1233
01:52:55.360 --> 01:53:00.880
Um so you you only need to do you don't need to do you don't need to check all 1,00 knobs.

s1234
01:53:00.880 --> 01:53:06.239
You only need to check a few steps and then you will know which flag is the best.

s1235
01:53:06.239 --> 01:53:10.480
Um so the trick is to use binary search or bisection to do this approach.

s1236
01:53:10.480 --> 01:53:12.880
Um yeah.

s1237
01:53:13.199 --> 01:53:13.599
Yeah.

s1238
01:53:13.599 --> 01:53:16.159
Any other questions?

s1239
01:53:17.119 --> 01:53:19.119
Yes.

s1240
01:53:19.599 --> 01:53:52.480
What are your thoughts on So your question was what do I think about asex like you know cerebras gro sanova I don't know even startups new chips they do design their own chips I I feel like so the problem of ASEX

s1241
01:53:52.480 --> 01:54:05.679
is is it AS6 or ASX or whatever the problem of specialized chips is the architecture itself needs to be hardcoded in some of the chips and that is the problem.

s1242
01:54:05.679 --> 01:54:16.960
If you hardcode some of the chips you know hard code the infra hardcode the architecture labs always like to change the architecture and so every single time when the lab changes the architecture

s1243
01:54:16.960 --> 01:54:18.960
you need to update the chip.

s1244
01:54:18.960 --> 01:54:30.719
Um but as a GPU the G the trick of GPUs is Nvidia has made it you know Nvidia, AMD, Intel whatever the GPU is extremely powerful because it has generalized

s1245
01:54:30.719 --> 01:54:36.800
asex inside of the GPU right the GPU is in fact a combination of ASEX.

s1246
01:54:36.800 --> 01:54:39.920
Um and the ASICH is just one large ASICH.

s1247
01:54:39.920 --> 01:54:46.960
Um so I think like in general a GPU is much better because you can customize what goes inside the GPU.

s1248
01:54:46.960 --> 01:54:49.760
you can disable stuff that goes inside the GPU and such.

s1249
01:54:49.760 --> 01:54:50.159
Yeah.

s1250
01:54:50.159 --> 01:55:00.159
So on so my view is I don't know I don't I don't want to say anything but like in general I don't think like you know previously as I mentioned

s1251
01:55:00.159 --> 01:55:18.560
you know hardware there is nowhere else to go you know we are at float four unless if the hardware providers invents float I don't know float zero then maybe we get four you know another four times faster but in general I think I think just people are focused too much on hardware and they have not looked

s1252
01:55:18.560 --> 01:55:22.159
that actually the biggest improvements is not hardware, it's software, right?

s1253
01:55:22.159 --> 01:55:27.119
Numerical precision, numerical precision was 32 times faster.

s1254
01:55:27.119 --> 01:55:29.599
Hardware is only three times faster, right?

s1255
01:55:29.599 --> 01:55:32.239
So hardware only contributed three times faster.

s1256
01:55:32.239 --> 01:55:37.360
Um oh actually d okay the die size you make the you make the GPU bigger, you get two times faster.

s1257
01:55:37.360 --> 01:55:38.320
That's that's kind of cheating.

s1258
01:55:38.320 --> 01:55:40.000
So I wouldn't really say that's improvement.

s1259
01:55:40.000 --> 01:55:44.639
Um but essentially if you make the hardware faster, you only get three times faster.

s1260
01:55:44.639 --> 01:55:48.320
Um so in my view hardware is probably overblown.

s1261
01:55:48.320 --> 01:55:50.480
you know hardware is actually not that important.

s1262
01:55:50.480 --> 01:56:01.679
The software was the trick that Nvidia, you know, Nvidia, AMD, Intel, all of these, you know, hardware providers, they banked on the fact that numerical precision was a trick and tensor,

s1263
01:56:01.679 --> 01:56:07.360
tensor, numerical precision, sparsity, you know, these software tricks.

s1264
01:56:07.360 --> 01:56:08.000
Um, okay.

s1265
01:56:08.000 --> 01:56:13.199
Well, tensor is not really software trick, but you know, a tensor is kind of an ASICH inside of the GPU.

s1266
01:56:13.199 --> 01:56:15.360
Um, and so like I feel like that's Yeah.

s1267
01:56:15.360 --> 01:56:18.000
So my view is I don't I don't really see a future for ASEX.

s1268
01:56:18.000 --> 01:56:18.719
That's my view.

s1269
01:56:18.719 --> 01:56:24.880
Um I think that ASEX are like instead, you know, to be honest, I'm actually quite surprised.

s1270
01:56:24.880 --> 01:56:28.960
We have lots of asset companies, but we have very few algorithm companies.

s1271
01:56:28.960 --> 01:56:32.960
Um and the reason why is because ASEX you can sell, right?

s1272
01:56:32.960 --> 01:56:34.719
Every single year you can upgrade.

s1273
01:56:34.719 --> 01:56:41.920
You know, this year you pay $1,000 to ASICH version one and the next year you have to upgrade, right?

s1274
01:56:41.920 --> 01:56:58.560
The problem with algorithms is algorithms is very hard to you know force the user or whatever to pay again and so that is why hardware is very popular because hardware is a very easy business model but for algorithms it gets more complicated right how are we going to monetize grading checkpointing

s1275
01:56:58.560 --> 01:57:10.400
I don't know right that's very hard um so but the main point is the large labs themselves I think like open air announced collaboration broadcom and suras whatever you know each lab themselves

s1276
01:57:10.400 --> 01:57:14.480
are going to the hardware provider and designing the chip with them.

s1277
01:57:14.480 --> 01:57:22.480
Um, so my view is like maybe we'll have more of these like collaboration approaches, but I feel like standalone standalone assets, I don't think they're going to last.

s1278
01:57:22.480 --> 01:57:26.800
Um, yeah, that's my take, I guess.

s1279
01:57:26.800 --> 01:57:28.719
Any other questions?

s1280
01:57:28.719 --> 01:57:30.719
Yes.

s1281
01:57:46.000 --> 01:57:49.599
Oh, you mean what are the types of kernels or

s1282
01:57:55.440 --> 01:57:56.000
Oh, okay.

s1283
01:57:56.000 --> 01:57:56.239
Okay.

s1284
01:57:56.239 --> 01:57:56.560
Okay.

s1285
01:57:56.560 --> 01:58:03.440
Um, so the question was what are the you know what are the changes for kernels or optimizations or stuff that is like interesting I guess for kernels.

s1286
01:58:03.440 --> 01:58:12.560
Um so most kernels when you write kernels the majority of them are are focused on memory movement reduction.

s1287
01:58:12.560 --> 01:58:14.560
How do we reduce memory movement?

s1288
01:58:14.560 --> 01:58:15.840
That's the majority of kernels.

s1289
01:58:15.840 --> 01:58:26.400
Um for example there's there's a trick called um you know there's a trick called fuse cross entropy loss where instead of making instead of the last layer of cont instead of materializing

s1290
01:58:26.400 --> 01:58:32.480
the full logs there is a trick you can do it in batches right you can do rowby row materialization.

s1291
01:58:32.480 --> 01:58:39.040
Um and so this will reduce memory by like a lot by like I don't know 10 GB or something if you have long context or even more.

s1292
01:58:39.040 --> 01:58:40.239
Um that's one way.

s1293
01:58:40.239 --> 01:58:53.920
Um the other kernels most kernels are called kernel fusion where you you have this like long pietorch function and all you do is you just write one kernel to do this whole pietorch function

s1294
01:58:53.920 --> 01:58:56.000
and torch compile will do this for you.

s1295
01:58:56.000 --> 01:59:10.159
So torch compile is very very good at doing kernel fusion right you give torch compile a function it will write a kernel a try kernel or whatever kernel and it will just fuse everything um it's very very effective for that um but I think in general

s1296
01:59:10.159 --> 01:59:22.480
kernels are just reducing memory movement um and so like I you know to be honest I don't really like to call it kernels most algorithms so most algorithms you either make training faster

s1297
01:59:22.480 --> 01:59:28.480
or reduce memory usage um but kernels in my view kernels is reduce memory movement.

s1298
01:59:28.480 --> 01:59:33.520
Um, and so most kernels is just memory movement, you know, memory movement optimization, right?

s1299
01:59:33.520 --> 01:59:36.639
How do we how do we use the caching structure of the GPUs?

s1300
01:59:36.639 --> 01:59:41.520
Um, you know, how do we not load the same variable twice or three times or whatever?

s1301
01:59:41.520 --> 01:59:46.400
Um, yeah, I'm not sure if that answer your question, but next reinforcement learning.

s1302
01:59:46.400 --> 01:59:49.679
Um, and after this will be reward hacking the agents.

s1303
01:59:49.679 --> 01:59:56.639
Um, so as a primer to I'm assuming most people know, do most people know reinforcement learning or do I need to prime people?

s1304
01:59:56.639 --> 01:59:59.599
Okay, I'll give a very fast primer for reinforcement learning.

s1305
01:59:59.599 --> 02:00:02.080
Okay, fast primer for reinforcement learning.

s1306
02:00:02.080 --> 02:00:04.960
Um, what is reinforcement learning?

s1307
02:00:04.960 --> 02:00:07.840
You have this environment such as this Pac-Man game.

s1308
02:00:07.840 --> 02:00:13.840
Um, and your goal is as the player, you know, to maximize reward.

s1309
02:00:13.840 --> 02:00:15.440
You want to eat all of the cookies, right?

s1310
02:00:15.440 --> 02:00:19.199
You want to eat all of the cookies, but also escape away from your the monsters.

s1311
02:00:19.199 --> 02:00:20.639
I don't actually know what they're called.

s1312
02:00:20.639 --> 02:00:23.119
Enemies, monsters, whatever, whatever they're called.

s1313
02:00:23.119 --> 02:00:27.920
Um, and your goal as Pac-Man is you want to maximize the amount of cookies that you eat.

s1314
02:00:27.920 --> 02:00:30.480
Um, and that is your reward.

s1315
02:00:30.480 --> 02:00:32.719
The reward is the cookies.

s1316
02:00:32.719 --> 02:00:36.400
Um, and the action is whether you go up, left, down, or right.

s1317
02:00:36.400 --> 02:00:40.000
Um, and the environment is the game.

s1318
02:00:40.000 --> 02:00:50.880
Another good example, you know, another way I like to explain reinforcement learning is the goal of reinforcement learning is you want to have more good and less bad um during training.

s1319
02:00:50.880 --> 02:00:55.920
Um so for example, at the very beginning of training, you ask the model what is 2 plus two?

s1320
02:00:55.920 --> 02:00:58.159
Um the answer is clearly four.

s1321
02:00:58.159 --> 02:01:02.880
Um but when the model starts training, it will be very dumb.

s1322
02:01:02.880 --> 02:01:03.760
It will be very bad.

s1323
02:01:03.760 --> 02:01:10.239
it will see B, you know, the model would just say B, D, cat, dog, house, mouse, whatever.

s1324
02:01:10.239 --> 02:01:19.199
Um, and the trick is for all of the bad responses, you want to decrease, you don't want you want to like negatively reward this or penalize it.

s1325
02:01:19.199 --> 02:01:25.760
You want to penalize the model if it says something bad and you want to increase the reward if it says the correct answer.

s1326
02:01:25.760 --> 02:01:28.320
Um, so that is the trick of reinforcement learning.

s1327
02:01:28.320 --> 02:01:32.239
You just want more good answers, less bad answers.

s1328
02:01:32.239 --> 02:01:35.280
Um, and if it's like, you know, very close to the correct answer.

s1329
02:01:35.280 --> 02:01:37.760
So, you know, three is very close to the correct answer.

s1330
02:01:37.760 --> 02:01:42.719
Um, you want to reward, you want to negatively reward this a little bit less, right?

s1331
02:01:42.719 --> 02:01:52.080
Because three is much closer to four than B or D. Um, so if you do B or D, you want to negatively reward it massively.

s1332
02:01:52.880 --> 02:01:57.360
In reinforcement learning, the trick is you have a verification system, right?

s1333
02:01:57.360 --> 02:02:01.199
You have a verifier to verify if the model is doing good or bad.

s1334
02:02:01.199 --> 02:02:04.239
Um so you'll call the model many many many times.

s1335
02:02:04.239 --> 02:02:08.000
Um and each of these examples you give a verification number right?

s1336
02:02:08.000 --> 02:02:13.280
So for example um the first example is very good so you give it a plus 10 score.

s1337
02:02:13.280 --> 02:02:17.760
Um the next example is like okay so you give it a minus5 score.

s1338
02:02:17.760 --> 02:02:22.880
Um and then the last example is very bad so you give it a minus 100 score.

s1339
02:02:22.880 --> 02:02:28.639
Um and reinforcement learning allows you to assign scores to each of those answers and questions.

s1340
02:02:28.639 --> 02:02:32.080
Um and so that's kind of reinforcement learning verifies.

s1341
02:02:32.080 --> 02:02:38.639
Um and the trick of reinforcement learning is my favorite phrase is patience is all you need.

s1342
02:02:38.639 --> 02:02:44.400
Um at the very beginning of training your model will do very bad, right?

s1343
02:02:44.400 --> 02:02:48.639
Your your reward will be 0000 you know 0000.

s1344
02:02:48.639 --> 02:02:52.400
You wait for a very long time and then you will get the correct answer.

s1345
02:02:52.400 --> 02:02:52.639
Right?

s1346
02:02:52.639 --> 02:02:56.080
So for example this example you ask the model what is 2 plus2 right?

s1347
02:02:56.080 --> 02:02:57.440
You start pre-training the model.

s1348
02:02:57.440 --> 02:03:04.159
you start pre-training the model, the model doesn't know what is two plus two, but after 10 years it will say four.

s1349
02:03:04.159 --> 02:03:06.639
Um, okay, obviously not 10 years, I'm just exaggerating.

s1350
02:03:06.639 --> 02:03:10.480
But after 10 years, you wait 10 years, the model will then say four.

s1351
02:03:10.480 --> 02:03:17.840
Um, and that is why my favorite phrase is luck is all you need for reinforcement learning.

s1352
02:03:17.840 --> 02:03:21.280
You know, maybe by chance you will get four very quickly.

s1353
02:03:21.280 --> 02:03:28.080
Um but you know maybe you just have to wait and wait and wait and wait for eternity until reinforcement learning works.

s1354
02:03:28.080 --> 02:03:38.159
Um and so in general your reward will be zero for a very long time and then you will get you know you will increase reward um after the zero

s1355
02:03:38.800 --> 02:03:51.840
and you know for reinforcement learning there is a very simple algorithm for reinforcement learning and the trick of reinforcement learning is remember you know the final answer you want to you for example

s1356
02:03:51.840 --> 02:04:00.480
you know what is 2 plus two you know the answer is four but the problem is you don't know what is the reasoning trace, you know, did was the reasoning trace good or bad?

s1357
02:04:00.480 --> 02:04:05.679
So, for example, this example is, you know, to tell the model to create a fast matrix multiplication algorithm.

s1358
02:04:05.679 --> 02:04:11.840
Um, and the trick is if the answer is right, you reward every single line as plus 10 score.

s1359
02:04:11.840 --> 02:04:18.480
Um, and if it's wrong, you reward every single score as minus 100.

s1360
02:04:18.480 --> 02:04:27.199
Um, and you know, Andre said, you know, in a Darkash podcast, reinforcement learning is kind of like sucking supervision bits through a straw.

s1361
02:04:27.199 --> 02:04:29.440
um you know we actually have stickers for them if you like.

s1362
02:04:29.440 --> 02:04:32.800
Um so you can get one of your stickers which we can distribute at the end.

s1363
02:04:32.800 --> 02:04:40.960
Um so Audrey's quote is this um you know and the main point is reinforcement learning is terrible but everything else is even worse.

s1364
02:04:40.960 --> 02:04:46.080
Um and so like you know reinforcement learning is the only tool we currently have that just works.

s1365
02:04:46.080 --> 02:04:48.239
It works but it's not very efficient.

s1366
02:04:48.239 --> 02:04:52.320
Um and okay actually okay that's the next section.

s1367
02:04:52.320 --> 02:04:55.199
Um but the main point is okay that's a reinforcement learning primer.

s1368
02:04:55.199 --> 02:04:59.760
Um I guess does anyone have questions on reinforcement learning primer?

s1369
02:04:59.760 --> 02:05:02.880
No. Okay, I'll skip to Okay, one question.

s1370
02:05:02.880 --> 02:05:04.880
Yes.

s1371
02:05:07.920 --> 02:05:09.840
I will mention that in the next section.

s1372
02:05:09.840 --> 02:05:14.480
Um there is there is like you know better RL methods.

s1373
02:05:14.480 --> 02:05:17.920
Um but in general reinforcement learning seems to do very well for now.

s1374
02:05:17.920 --> 02:05:20.639
Last I think this is the last topic or maybe not.

s1375
02:05:20.639 --> 02:05:23.040
Reward hacking and agents.

s1376
02:05:23.040 --> 02:05:24.719
The most fun one I guess.

s1377
02:05:24.719 --> 02:05:35.520
Um so okay for reinforcement learning reinforcement learning can only work if the probability of a good answer is more than zero.

s1378
02:05:35.520 --> 02:05:38.960
If it is less than zero reinforcement learning will never work.

s1379
02:05:38.960 --> 02:05:42.000
So that is a fun that is a constraint of reinforcement learning.

s1380
02:05:42.000 --> 02:05:44.639
The probability of a good answer must be more than zero.

s1381
02:05:44.639 --> 02:05:46.080
It can never be zero.

s1382
02:05:46.080 --> 02:05:51.040
Um and there are many many many problems of reinforcement learning not working.

s1383
02:05:51.040 --> 02:05:54.000
You know the formatting could be wrong.

s1384
02:05:54.000 --> 02:05:56.560
you know, you need to do some sort of priming or warm up.

s1385
02:05:56.560 --> 02:06:03.520
So, you have to do like some sort of trick to teach the model a little bit about, you know, about the thing that you're trying to maximize.

s1386
02:06:03.520 --> 02:06:06.719
Um, you have to do supervised finetuning.

s1387
02:06:06.719 --> 02:06:12.960
So, one of the tricks of reinforcement learning is you actually need to do SFT or fine-tuning to make the model not dumb, right?

s1388
02:06:12.960 --> 02:06:16.960
To make the probability of zero not zero, the probability of a good answer not zero.

s1389
02:06:16.960 --> 02:06:19.040
Um, you need to do good pre-training.

s1390
02:06:19.040 --> 02:06:27.119
Um and then the other problem is that you know during reinforcement learning it's just way too out of distribution that reinforcement learning is just very bad.

s1391
02:06:27.119 --> 02:06:49.040
Um so there are many many problems of reinforcement learning and I think we just you know for the trajectories reinforcement learning can assign incorrect rewards to the trajectory right remember the simple trick of reinforcement learning is we assign the reward to every single line as the same number right either this is good or this is bad

s1392
02:06:49.040 --> 02:06:56.960
and this is not good because why right you ask the model I need to find what is 2 plus two the answer is correct Right?

s1393
02:06:56.960 --> 02:06:58.239
The answer is four.

s1394
02:06:58.239 --> 02:06:59.760
The model says it's four.

s1395
02:06:59.760 --> 02:07:03.440
So you reward this whole thinking trace as plus 10.

s1396
02:07:03.440 --> 02:07:09.119
But this is wrong because as you can see in the thinking trace it says 2 plus 2 is equal to 10.

s1397
02:07:09.119 --> 02:07:18.960
Imagine you know in all of training because the trick of reinforcement learning is we just literally assign 10 to every single line or minus 100 to every single line.

s1398
02:07:18.960 --> 02:07:22.400
We missed this bad you know bad thing.

s1399
02:07:22.400 --> 02:07:35.360
Um so you can imagine when we keep training the model the model might hack or do reward hacking or you know make gibberish it will do gibberish in between do some do some sort of like new machine language

s1400
02:07:35.360 --> 02:07:39.679
which we can't read and it will assign high score to that.

s1401
02:07:39.679 --> 02:07:49.440
Um and so this is a very big problem of reinforcement learning and the way to solve this or fix this is something called process supervision.

s1402
02:07:49.440 --> 02:08:05.920
Um and process supervision what you do is you manually check every single line not you don't just assign plus 10 to the final you know the answer is correct right the answer is correct plus 10 assign every single line as plus 10 you don't do this instead what you do is you assign every single line as a

s1403
02:08:05.920 --> 02:08:20.159
different number right you assign some lines as plus 30 some lines is plus zero whatever the bad lines is minus 100 right this works very very well um unfortunately

s1404
02:08:20.159 --> 02:08:40.320
process vision cannot scale and it's extremely expensive to do right who's going to label this it's the you know the humans I guess right we have to label this data right we have to manually label for the labs I guess that's why labs sometimes like you know they go to scale call whatever right they ask people to

s1405
02:08:40.320 --> 02:08:49.119
label the data you know is this good is this bad is this good is this bad um and so on um but the trick is you can also use a language model, right?

s1406
02:08:49.119 --> 02:08:50.880
You can use LLM as a judge.

s1407
02:08:50.880 --> 02:08:56.480
You can you can call a language model to label every single line.

s1408
02:08:56.480 --> 02:09:00.639
And you know, my view is like, you know, large labs are going to be doing this process more.

s1409
02:09:00.639 --> 02:09:05.040
They will call their own model iteratively to re-review itself.

s1410
02:09:05.040 --> 02:09:09.199
Um, and that is one way their view is they can reach AGI, right?

s1411
02:09:09.199 --> 02:09:11.440
Just by by re-reviewing itself, right?

s1412
02:09:11.440 --> 02:09:20.320
re-evaluating itself, re-checking, doing doing you know automatic LLM as a judge process supervision something like this.

s1413
02:09:20.320 --> 02:09:38.159
Um but remember there is a problem because even if you do process supervision the model you are using the same model to evaluate the model right the same problem as we bench pro right su bench pro you use the LLM as a verifier to verify the LLM which is definitely not good

s1414
02:09:38.159 --> 02:09:51.760
um and the reason why is because you can do reward hacking um a very good example of reward hacking is your model starts cheating um so for example when you want to make a fast matrix multiplication

s1415
02:09:51.760 --> 02:09:52.880
algorithm.

s1416
02:09:52.880 --> 02:09:55.119
All it does is it deletes the timer.

s1417
02:09:55.119 --> 02:09:59.199
Um right remember you give the goal to max to reduce the time.

s1418
02:09:59.199 --> 02:09:59.520
Right?

s1419
02:09:59.520 --> 02:10:02.320
Reduce the time of the matrix multiplication algorithm.

s1420
02:10:02.320 --> 02:10:04.880
Um so all it will do is just delete the timer.

s1421
02:10:04.880 --> 02:10:06.239
Let's delete the timer.

s1422
02:10:06.239 --> 02:10:10.320
Set the timer to be zero and then there we maximize a reward.

s1423
02:10:10.320 --> 02:10:12.719
Um obviously this is not correct, right?

s1424
02:10:12.719 --> 02:10:15.840
Because the trick is you also have a correctness check, right?

s1425
02:10:15.840 --> 02:10:18.639
You check if the matrix multiplication is actually correct.

s1426
02:10:18.639 --> 02:10:24.800
Um but there is another way the model will edit your two matrices to be just zero.

s1427
02:10:24.800 --> 02:10:26.960
Um and what is 0 time 0?

s1428
02:10:26.960 --> 02:10:27.520
Zero.

s1429
02:10:27.520 --> 02:10:30.000
Um and so the correctness checks also fail.

s1430
02:10:30.000 --> 02:10:43.360
Um and so reward hacking becomes a very very big problem because these models can cheat and do special tricks to go around your actual model um your intent of the reward function.

s1431
02:10:43.840 --> 02:10:48.400
Another very problematic example is it's not just about reward hacking.

s1432
02:10:48.400 --> 02:10:50.639
It can actually destroy your computer.

s1433
02:10:50.639 --> 02:10:51.599
Right?

s1434
02:10:51.599 --> 02:11:02.079
By bad luck your model might output you know some sort of corruption methodology you know deleting you know doing rm-rf on your entire computer and bye-bye your computer's dead.

s1435
02:11:02.079 --> 02:11:04.719
Um and so like you know sometimes this also does happen.

s1436
02:11:04.719 --> 02:11:14.079
Um so it's not just reward hacking also trust of your tool cause you know trust of whether the model is actually doing good or bad is also a very big problem

s1437
02:11:14.079 --> 02:11:24.079
and remember this plot that I showed you know if you include GPD 5.6 cheating on the benchmarks you know looking at the answer you know remember the previously su bench

s1438
02:11:24.079 --> 02:11:32.880
uh bench pro and deep show that models also cheat by looking at the final answer you know you can see that with GBD 5.6 If you cheat, it does very well.

s1439
02:11:32.880 --> 02:11:38.000
But if you remove the cheating examples, it does, you know, within trend.

s1440
02:11:38.239 --> 02:11:44.159
And then, you know, maybe you might be thinking, oh, this reward hacking thing is like, oh, it's like very rare, you know, very rare.

s1441
02:11:44.159 --> 02:11:45.840
It's not going to happen in real world.

s1442
02:11:45.840 --> 02:11:54.960
Um, well, GLM 5.2 during its training methodology, they specifically mentioned they have this new methodology for reinforcement learning called anti-hacking.

s1443
02:11:54.960 --> 02:12:00.560
Um, so GLM 5.2 introduced a method to stop, you know, reward hacking.

s1444
02:12:00.560 --> 02:12:04.320
Um and what they do is they added a link checker.

s1445
02:12:04.320 --> 02:12:10.880
Um so remember previously we mentioned how SweetBench Pro um the model will cheat and look at the answer.

s1446
02:12:10.880 --> 02:12:14.320
Um and so what GLM did is they had this check.

s1447
02:12:14.320 --> 02:12:18.480
Um so during reinforcement learning they will check every single tool call you make.

s1448
02:12:18.480 --> 02:12:25.040
Um and if the website if the website went to the answer you would stop that from happening.

s1449
02:12:25.040 --> 02:12:30.400
Um and so like GLM essentially outed this like you know filtering system for the entire reinforcement learning process.

s1450
02:12:30.400 --> 02:12:36.960
Um and you know according to them it worked very well and remember this plot about cheating examples.

s1451
02:12:36.960 --> 02:12:40.320
Um you know opus it seems like claude's models like to always cheat.

s1452
02:12:40.320 --> 02:12:42.239
Um and Jubet's models don't like to cheat.

s1453
02:12:42.239 --> 02:12:52.239
Um but the main takeaway is models will cheat because you are you are telling it you know like you know I want to maximize reward AB CDE EFG.

s1454
02:12:52.239 --> 02:12:57.040
Um and so the model will it will maximize it but it won't actually follow your intent.

s1455
02:12:57.040 --> 02:12:59.119
Um so you have to be very careful on this.

s1456
02:12:59.119 --> 02:13:06.480
Um in fact for GBD 5.1 during its training OpenAI mentioned that they had something called calculator hacking.

s1457
02:13:06.480 --> 02:13:16.320
Um and so in GBD 5.1 when they were training um they wanted to reward web tool use right so like you want to reward the model to use the web tool.

s1458
02:13:16.320 --> 02:13:21.280
Um but instead it didn't use the web tool it used the calculator to fake the web tool.

s1459
02:13:21.280 --> 02:13:24.960
Um and so during the training of GBD 5.1 this happened.

s1460
02:13:24.960 --> 02:13:28.320
Um and so like you know there's many many many many problems.

s1461
02:13:28.320 --> 02:13:31.199
I think they show yeah they showed calculator hacking.

s1462
02:13:31.199 --> 02:13:33.840
You know you lie about which tool you used.

s1463
02:13:33.840 --> 02:13:37.840
Um you know you conceal uncertainty you make facts up.

s1464
02:13:37.840 --> 02:13:41.920
Um so there's many many many problems with um reward hacking.

s1465
02:13:41.920 --> 02:13:43.520
And this is not fake right?

s1466
02:13:43.520 --> 02:13:46.639
So reward hacking is already in large labs training runs right?

s1467
02:13:46.639 --> 02:13:51.360
This is just 5.1. Um I don't think so they mentioned GB 5.2 or whatever.

s1468
02:13:51.360 --> 02:14:01.760
Yeah, but in general they showed that you know this thing does happen in real world um you know I don't know if you guys know GPU mode um but GPU mode does you know this leaderboard

s1469
02:14:01.760 --> 02:14:11.920
um for you know making faster kernels so if you do want to write your own kernels definitely post on GPU modes hackathon challenges um they're very very helpful and very useful um but you know someone

s1470
02:14:11.920 --> 02:14:22.000
managed to hack reward hack the GPU mode kernel competition um and remember in the Matrix multiplication example.

s1471
02:14:22.000 --> 02:14:35.599
There are two there are two there are two checks that we need to do right make the matrix multiplication algorithm faster but also it needs to be correct right there are two checks the correctness check and the timing check.

s1472
02:14:35.599 --> 02:14:40.320
Um and GPU mode also had timing check.

s1473
02:14:40.320 --> 02:14:50.880
Um and so what do you think the model did when the model the model knew the model actually knew that it was being evaluated on the correctness check right it learned

s1474
02:14:50.880 --> 02:15:03.040
oh I'm being evaluated on the correctness check I will now make correctness correct right so it will output the correct kernel and then the model knew that it was getting timed

s1475
02:15:03.040 --> 02:15:12.079
and what it what did it do it just it just did the algorithm once and then saved it and it skipped all the another 15 um you know tests.

s1476
02:15:12.079 --> 02:15:13.599
Um and so that's what the model did.

s1477
02:15:13.599 --> 02:15:20.719
So essentially the model learned the model learned that there were two tests, the correctness check and the timing check.

s1478
02:15:20.719 --> 02:15:27.840
And the model only did the correctness check correctly and then once it went into the regime of timing, it cheated.

s1479
02:15:27.840 --> 02:15:30.159
Um to be honest, it's actually quite scary.

s1480
02:15:30.159 --> 02:15:35.679
So essentially the model learned that you're doing these tests and the model actually knows you're doing the benchmarks.

s1481
02:15:35.679 --> 02:15:37.920
Um and so this is actually very interesting.

s1482
02:15:37.920 --> 02:15:42.079
Um and you know, oh yeah, this is this is more an you know, larger example.

s1483
02:15:42.079 --> 02:15:46.400
The correctness check was fine, but the timing check it cheated.

s1484
02:15:46.400 --> 02:15:51.920
Um and all it did is it launch, you know, there was supposed to be 15 calls in the first call.

s1485
02:15:51.920 --> 02:15:56.000
In the first core, it did all 15 of the entire process, right?

s1486
02:15:56.000 --> 02:15:57.520
It did all of the 15 runs.

s1487
02:15:57.520 --> 02:16:01.040
Um and then core two to 15, it just did a Python dictionary look up.

s1488
02:16:01.040 --> 02:16:02.800
Um, yeah.

s1489
02:16:02.800 --> 02:16:04.320
I don't know if you know about this.

s1490
02:16:04.320 --> 02:16:08.800
Reminds me of Volkswagen where they initi.

s1491
02:16:08.880 --> 02:16:10.639
Yes, I someone did tell

s1492
02:16:10.639 --> 02:16:11.599
someone told me about it.

s1493
02:16:11.599 --> 02:16:12.159
Um,

s1494
02:16:12.159 --> 02:16:13.119
this is very similar.

s1495
02:16:13.119 --> 02:16:14.800
It's like, oh, I'm not doing this.

s1496
02:16:14.800 --> 02:16:17.599
Turn this off and

s1497
02:16:18.000 --> 02:16:18.560
Yeah, exactly.

s1498
02:16:18.560 --> 02:16:20.960
So, like, you know, it's not just models, I guess, that cheat.

s1499
02:16:20.960 --> 02:16:22.639
Even humans cheat, I guess.

s1500
02:16:22.639 --> 02:16:23.040
Yes.

s1501
02:16:23.040 --> 02:16:24.639
But I think it's called Goodart's law.

s1502
02:16:24.639 --> 02:16:25.040
That's the one.

s1503
02:16:25.040 --> 02:16:29.520
Like, if you have a benchmark, then the benchmark becomes Is it good law?

s1504
02:16:29.520 --> 02:16:30.320
I don't remember.

s1505
02:16:30.320 --> 02:16:30.719
Yes.

s1506
02:16:30.719 --> 02:16:31.119
Okay.

s1507
02:16:31.119 --> 02:16:31.280
Yeah.

s1508
02:16:31.280 --> 02:16:35.359
The benchmark essentially becomes useless because people just cheat to maximize reward.

s1509
02:16:35.359 --> 02:16:38.000
Um yes, I guess humans also cheat.

s1510
02:16:38.000 --> 02:16:39.599
Um yeah.

s1511
02:16:39.599 --> 02:16:40.800
Okay.

s1512
02:16:40.800 --> 02:16:51.679
Oh, my favorite example is um so on other labs, you know, you see on Twitter, on wherever, they say they made kernels 10 times faster.

s1513
02:16:51.679 --> 02:16:54.719
Um no, no, no, that's not correct.

s1514
02:16:54.719 --> 02:16:56.800
They did not make kernels 10 times faster.

s1515
02:16:56.800 --> 02:17:01.519
In fact, if you look through the code, they have no, you know, no ops, so no operations.

s1516
02:17:01.519 --> 02:17:03.200
They also edit the timer.

s1517
02:17:03.200 --> 02:17:13.760
You know, they, as I literally described, you know, I described, they, you know, over here, um, you know, they edit the timer, they made matrices go to zero, they cheated.

s1518
02:17:13.760 --> 02:17:17.439
Um, and so like, you know, this actually happened in real world.

s1519
02:17:17.439 --> 02:17:23.679
So some, you know, some of the labs, they published papers claiming that they made kernels 10 times faster.

s1520
02:17:23.679 --> 02:17:29.040
But actually if you read through the code and the examples they these examples all cheated.

s1521
02:17:29.040 --> 02:17:34.080
Um and so you know they you know this is not very good in terms of you know reward hacking.

s1522
02:17:34.080 --> 02:17:36.000
You know reward hacking is a very big problem.

s1523
02:17:36.000 --> 02:17:47.280
Um and you know for example what some of the examples of kernel reward hacking you know not generating real CUDA code instead it cause or some sort of like you know already written

s1524
02:17:47.280 --> 02:17:48.160
system.

s1525
02:17:48.160 --> 02:17:53.439
um you have no up kernels which is essentially making the you know making the A and B matrix just zero.

s1526
02:17:53.439 --> 02:17:55.679
All it does is just doesn't do anything, right?

s1527
02:17:55.679 --> 02:17:57.519
It just the kernel is empty.

s1528
02:17:57.519 --> 02:18:02.639
Um, and you have like memory reuse, so you reuse the same answer over and over again.

s1529
02:18:02.639 --> 02:18:05.200
Um, you have timing synchronization issues.

s1530
02:18:05.200 --> 02:18:07.040
So that's cheating on the timer.

s1531
02:18:07.040 --> 02:18:19.040
Um and my view is like you know if you do publish faster kernels or faster you know matrix if you think that your AI agent has made kernels 10 times faster please verify

s1532
02:18:19.040 --> 02:18:31.200
you know please look through the code before publishing because it is a very it's not a very good look um and so and also the biggest issue that I feel like people are getting forgetting is

s1533
02:18:31.200 --> 02:18:42.319
you know you made kernels 10 times faster you made matrix multiplication 10 times faster There is a theoretical limit for matrix multiplication, right?

s1534
02:18:42.319 --> 02:18:48.880
Matrix multiplication, you know, it's not you can't make a faster because there's mathematical limits on how to make a faster, right?

s1535
02:18:48.880 --> 02:18:54.319
And so like, you know, matrix multiplication at the very very very olden times, you know, it's O of N cubed.

s1536
02:18:54.319 --> 02:18:58.719
You know, every single time researchers have make it faster and faster and faster and faster and faster.

s1537
02:18:58.719 --> 02:19:03.120
You know, it's now O of N to the^ of 2.371339, I guess.

s1538
02:19:03.120 --> 02:19:08.240
you know researchers every single year are trying to like make this number smaller and smaller and smaller and smaller.

s1539
02:19:08.240 --> 02:19:14.080
Um you know I guess like you know 1 1552 to 1339 is not that small you know not that big I guess.

s1540
02:19:14.080 --> 02:19:27.599
Um but you know they're having progress but the main point is you know these researchers you know they show with mathematical limits you cannot go faster than this and so how can you do reward hacking that is even faster than that.

s1541
02:19:27.599 --> 02:19:36.240
Um and so like the fundamental point is please verify you know to like the people who do research papers and stuff like that please confirm your model is not reward hacking.

s1542
02:19:36.240 --> 02:19:38.319
It is a very big big big problem.

s1543
02:19:38.319 --> 02:19:41.359
Um and you can see oh I think I only had one plot.

s1544
02:19:41.359 --> 02:19:46.559
Um but yes in general please do not do please check your I guess models.

s1545
02:19:46.559 --> 02:19:49.600
Um I guess that's all for the talk.

s1546
02:19:49.600 --> 02:19:52.560
Um you know yeah thank you everyone for coming.

s1547
02:19:52.560 --> 02:19:54.319
Oh more questions as well.

s1548
02:19:54.319 --> 02:19:55.359
Um okay thank you.

s1549
02:19:55.359 --> 02:19:55.920
Thank you.

s1550
02:19:55.920 --> 02:19:57.196
We also have

s1551
02:19:57.196 --> 02:19:58.240
[applause]

s1552
02:19:58.240 --> 02:19:58.880
Oh, yes.

s1553
02:19:58.880 --> 02:20:05.359
We have a whole bunch of stickers that you can take in the box over there and some pins and stuff.

s1554
02:20:16.847 --> 02:20:18.847
[music]