WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=vrDvatGtIxs

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/vrDvatGtIxs.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:12.400 --> 00:00:14.559
Hi everyone, I'm Tis.

s2
00:00:14.559 --> 00:00:19.039
Uh, so I'm going to be explaining how we make models three times faster with auto research.

s3
00:00:19.039 --> 00:00:29.359
Uh, so previous to this, uh, I actually used to do GPU mining in my dorm room with 1080 Ti all the way up to working at Tesla on inference optimization for Tesla AI.

s4
00:00:29.920 --> 00:00:31.760
Uh but first what is auto research?

s5
00:00:31.760 --> 00:00:46.239
So auto research is this framework from Andre Kapathy where uh you basically set up a framework for an agent to move towards a goal that you define uh and all you have to do basically is say at the high level what you want it to do and it will try things as it goes and move back and

s6
00:00:46.239 --> 00:00:49.360
forth uh towards that goal.

s7
00:00:49.600 --> 00:00:51.280
In actuality it's really just a while loop.

s8
00:00:51.280 --> 00:00:53.039
The agent proposes a solution.

s9
00:00:53.039 --> 00:00:57.680
You have a setup to to define what's correct benchmark it for us.

s10
00:00:57.680 --> 00:01:02.719
uh and then you keep or revert that and you do this in a loop until your goal is met.

s11
00:01:02.719 --> 00:01:05.360
And so this is very well aligned to GPU kernels.

s12
00:01:05.360 --> 00:01:08.560
Uh so if you don't know what a GPU kernel is, it's basically a low-level operator.

s13
00:01:08.560 --> 00:01:10.880
And in a Nvidia GPU, this is a CUDA kernel.

s14
00:01:10.880 --> 00:01:19.840
Uh and this is um an operator that's used by the GPU to operate like millions of times in parallel, for example, like a matrix multiply or an expert computation.

s15
00:01:20.720 --> 00:01:22.880
And why are GPUs such a good fit for auto research?

s16
00:01:22.880 --> 00:01:24.479
It's because they're super verifiable.

s17
00:01:24.479 --> 00:01:29.840
you can verify them for correctness and speed and that's basically all you need for your auto research framework.

s18
00:01:30.080 --> 00:01:32.560
Uh so in actuality there are some caveats here.

s19
00:01:32.560 --> 00:01:41.040
Um the auto research framework is really good for like picking block sizes and these tiny parameters but they're also still really bad at the high level idea like seeing like

s20
00:01:41.040 --> 00:01:43.600
I want to use this GPU and I actually want to pipeline it.

s21
00:01:43.600 --> 00:01:45.200
It's not going to come up with these groundbreaking ideas.

s22
00:01:45.200 --> 00:01:46.479
So it's still up to the human to do that.

s23
00:01:46.479 --> 00:01:51.600
But the actual implementation is very straightforward once you once you have the idea laid out.

s24
00:01:51.600 --> 00:01:55.119
So it is still your job to have good ideas is what I'm saying.

s25
00:01:55.119 --> 00:02:06.079
Uh and so the actual secret formula here is you have the good ideas auto research picks out the parameters and everything to verify that it actually works uh and go moves toward that verifiable goal of it being x times faster and

s26
00:02:06.079 --> 00:02:13.920
uh still correct and you mix that with billions of tokens of your favorite model and that results in kernels that beat hand tuning.

s27
00:02:13.920 --> 00:02:19.360
Uh so what are the actual things you care about when you're when you're when you're writing a custom kernel or you're having your agent write a custom kernel?

s28
00:02:19.360 --> 00:02:31.360
So the three main things you can have are a compute bottleneck uh a memory bottleneck or you just have excessive overhead from uh too many kernels being launched and you can do you can view these things with by profiling with a profiler like NSIS

s29
00:02:31.360 --> 00:02:34.400
for example which is Nvidia's profiler.

s30
00:02:34.400 --> 00:02:41.840
Uh, and so this this gra this page looks super daunting, but basically your job as a human is to look at the top here and be like this is dumb.

s31
00:02:41.840 --> 00:02:44.959
Uh, we are loading 32k chunks into context.

s32
00:02:44.959 --> 00:02:48.720
Uh, and we don't actually need to for this deepseek attention for example.

s33
00:02:48.720 --> 00:02:50.959
Uh, and we should only be doing it every 32k instead.

s34
00:02:50.959 --> 00:02:54.959
And so at a high level, all you have to be telling auto research is this top method is dumb.

s35
00:02:54.959 --> 00:02:56.000
Let's pipeline it instead.

s36
00:02:56.000 --> 00:03:02.560
And everything else like the sizing, the chunk sizing, the context chunks, that all should just be decided by auto research.

s37
00:03:03.040 --> 00:03:05.440
And so my problem is that I really love cheap GPUs.

s38
00:03:05.440 --> 00:03:11.680
And so that means like GPUs that don't have NVLink for example uh is example of like GPUs you can get for cheaper.

s39
00:03:11.680 --> 00:03:14.000
Uh but the problem is you don't actually have kernels off the shelf for those.

s40
00:03:14.000 --> 00:03:16.959
And so you have to come up with a auto research framework as well as a custom harness.

s41
00:03:16.959 --> 00:03:20.239
So what goes into the harness to make this really good.

s42
00:03:20.239 --> 00:03:23.920
Uh so one thing you really need to make sure your agent is aware of is the hardware.

s43
00:03:23.920 --> 00:03:28.480
And so on a B200 for example, you need to make sure it has context of uh the warps.

s44
00:03:28.480 --> 00:03:29.920
It has T-M TMA.

s45
00:03:29.920 --> 00:03:36.480
And so if you don't know what these are, these are just uh low-level operators that you have um on a specific hardware.

s46
00:03:36.480 --> 00:03:38.239
And this changes generation to generation.

s47
00:03:38.239 --> 00:03:40.879
Like H200 won't have T-M for example.

s48
00:03:40.879 --> 00:03:44.560
That's a new feature that came out with B200, which is why you need to have this in context.

s49
00:03:44.560 --> 00:03:50.720
Um and so this this basically is just like bunch of MD files you need to give so it has context.

s50
00:03:50.879 --> 00:03:53.200
Other thing you need to make sure your agent has context of is the model.

s51
00:03:53.200 --> 00:03:56.400
And so every new model like Deepseek Flash comes out with like new tricks.

s52
00:03:56.400 --> 00:04:02.000
like deepseek had two new attentions that was released in the deepseek flash for deepseek v4.

s53
00:04:02.000 --> 00:04:11.920
Uh so compressed sparse attention hierarchal compress and if you don't do this the model will 100% hallucinate uh the actual attention mechanism and you will get useless kernels.

s54
00:04:12.560 --> 00:04:15.680
Uh by far the biggest problem when you're doing this is going to be reward hacking.

s55
00:04:15.680 --> 00:04:23.120
And so if you were to tell your kernel engineer co-orker I need to make uh the GPU this GPU kernel faster.

s56
00:04:23.120 --> 00:04:30.000
uh it's obviously not going to your human co-workers is not going to go in and do some stuff that's going to make it slow like the endto-end model inference slower but

s57
00:04:30.000 --> 00:04:43.520
uh agents are not humans and they will do plenty of things to make it slower like they'll disable CUDA graphs which can make it 20 times slower and they might make that one kernel faster but make the whole like it's not a viable kernel because it's they're disabling a bunch of speed ups like CUDA graphs or only

s58
00:04:43.520 --> 00:04:51.759
testing on small context windows and so a lot of this is also just defining what not to do which is actually very important when you're doing frontier work that agents can't actually

s59
00:04:51.759 --> 00:04:55.120
easily with a one shot.

s60
00:04:56.479 --> 00:05:02.560
Uh, another reward hack is that some models just don't actually write the cute DSL you need uh when you're trying to write kernels.

s61
00:05:02.560 --> 00:05:04.639
And this is a common problem with enthropic models.

s62
00:05:04.639 --> 00:05:09.840
And so, yeah, I mean, Anthropic says what they say about uh nerfing models.

s63
00:05:09.840 --> 00:05:14.960
You can it's guess if it's I'm guessing if it's nerfing or not, but I would recommend using a different model.

s64
00:05:14.960 --> 00:05:17.199
Uh, and it won't always be faster everywhere actually.

s65
00:05:17.199 --> 00:05:26.880
So sometimes the kernels you come up with might only work well on like zero to 100k and then you need to go back to this the default kernel that you could you get from like a flash infer cutless.

s66
00:05:26.880 --> 00:05:34.080
Um and so and that's another thing to look out for is that your kernel isn't always just a swap in for all all workloads.

s67
00:05:34.400 --> 00:05:36.720
Uh but one of the great things is is that kernels compound.

s68
00:05:36.720 --> 00:05:44.880
So like if you make one for your sparse MLA for deepseek for example um you can get speed ups there and you just stack them on like that then plus NVFP4

s69
00:05:44.880 --> 00:05:55.440
fore uh you could do for us if we if you don't have NVL link you just keep stacking and stacking and stacking and then eventually you taper off at whatever the hardware limit is uh for your GPU and that's

s70
00:05:55.440 --> 00:06:01.919
uh some people call this like MFU which is like the actual theoretical max utilization from a GPU.

s71
00:06:01.919 --> 00:06:08.240
Uh and so to go even farther, if you have actually have bare metal access, your auto research framework can uh do very hacky things.

s72
00:06:08.240 --> 00:06:10.560
So hackers that have hacked with GPUs are probably going to like this.

s73
00:06:10.560 --> 00:06:22.240
You can uh tweak your BIOS settings, you can overclock the GPU, uh you can force like PCIe relaxing, all these little tweaks of like uh old school hackers used to do, but this can actually help with inference as well.

s74
00:06:22.240 --> 00:06:31.199
And so net on bare metal optimizations, you can get roughly 25% over like a virtualized setup you get from using a cloud provider.

s75
00:06:31.520 --> 00:06:36.400
Uh so once you've done that, you can combine all of the kernels you did as well as all of the hardware level hacks you did.

s76
00:06:36.400 --> 00:06:38.000
Uh you can get a 3x speed up.

s77
00:06:38.000 --> 00:06:41.919
And so I know this this might all sound like roses and flowers, but it's not actually the case.

s78
00:06:41.919 --> 00:06:45.199
Around 80% of the things that auto research is going to do are going to be bad.

s79
00:06:45.199 --> 00:06:50.240
Uh so it's important to remember while you're u like working on this that most things are going to be bad.

s80
00:06:50.240 --> 00:06:52.240
It's going to try to trick you all the time.

s81
00:06:52.240 --> 00:06:56.800
Uh but at the end you can actually get really good results from this.

s82
00:06:56.880 --> 00:06:58.080
with TLDDR.

s83
00:06:58.080 --> 00:07:00.800
Uh have better ideas than use auto research.

s84
00:07:00.800 --> 00:07:01.440
Super simple.

s85
00:07:01.440 --> 00:07:03.680
Simple, right?

s86
00:07:03.680 --> 00:07:06.000
Uh so it turns out you can actually get paid to do this.

s87
00:07:06.000 --> 00:07:11.039
Uh if you think this is cool, consider joining us and you can email me here.

s88
00:07:11.039 --> 00:07:13.360
Thanks, guys.