WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=pOvWgX7IJsc

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/pOvWgX7IJsc.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.800 --> 00:00:13.679
Hi everyone.

s3
00:00:13.679 --> 00:00:15.599
Uh, sorry it's a bit loud in here.

s4
00:00:15.599 --> 00:00:17.119
Was not expecting this.

s5
00:00:17.119 --> 00:00:18.640
Um, I'm Siman.

s6
00:00:18.640 --> 00:00:21.359
I'm a principal scientist at Together AI.

s7
00:00:21.359 --> 00:00:29.439
Um, I previously did my PhD in the Hazy Research Lab with Chris Ray at Stanford and I'm an incoming professor at Caltech.

s8
00:00:29.439 --> 00:00:42.079
Um I lead the frontier performance research team at together where we develop systems, frameworks and algorithms to extract as much performance as possible out of modern um AI hardware.

s9
00:00:42.079 --> 00:00:51.520
Today I want to share a little bit about our contributions towards simplifying the development of uh multi-GPU AI kernels.

s10
00:00:51.520 --> 00:01:01.440
A few years ago, um, GPU utilization used to be limited by poor intraGPU memory access and single GPU kernels.

s11
00:01:01.440 --> 00:01:15.840
But with significant investment in better kernels like flash attention, uh, memory efficient architectures like from deepseek, sparse attentions, mambas and so on, um, and better DSLs, we've sort of shifted the bottleneck

s12
00:01:15.840 --> 00:01:19.600
to multi-GPU communication.

s13
00:01:21.200 --> 00:01:27.600
During this talk, I'll start by telling you a little bit about why now, why GPU networking now.

s14
00:01:27.600 --> 00:01:30.880
Then I'll tell you about um the sort of problem space.

s15
00:01:30.880 --> 00:01:38.000
So what are the challenges in maximizing hardware utilization and development simplicity for multiGPU kernels.

s16
00:01:38.000 --> 00:01:45.119
Um three, we'll talk a little bit about the fundamentals be behind designing effective multiGGPU kernels.

s17
00:01:45.119 --> 00:01:52.000
Four, we'll look at whether frontier AI models can uh leverage these fundamental principles.

s18
00:01:52.000 --> 00:01:53.680
Do they understand them?

s19
00:01:53.680 --> 00:01:55.200
Can they reason about them?

s20
00:01:55.200 --> 00:01:58.399
Um, you know, in theory, these models are very good at reasoning.

s21
00:01:58.399 --> 00:02:10.319
Um, and then five, we'll talk through the results of these frontier models on a benchmark that we've developed called um parallel kernel bench for multiGPU kernel generation evaluation.

s22
00:02:12.080 --> 00:02:16.959
Okay, before we dive into those five parts, just basic preliminaries.

s23
00:02:16.959 --> 00:02:23.760
So, this is an Nvidia GPU, uh, an H100 GPU that you can see on the screen.

s24
00:02:23.760 --> 00:02:29.760
Um, I always like to help ground people in GPU kernels via looking at the hardware.

s25
00:02:29.760 --> 00:02:35.360
So, these rainbow colored dots are processors where actual compute is happening.

s26
00:02:35.360 --> 00:02:47.120
um all of the you know parallel threads are operating within one of those colored dots and there's typically you know 100 200 of them on modern AI GPUs.

s27
00:02:47.120 --> 00:02:57.200
Um around those processors you can see um some of the memory that these processors retrieve data so large weights activations from.

s28
00:02:57.200 --> 00:03:04.560
So these rectangles between the colored dots are an L2 cache slightly faster memory.

s29
00:03:04.560 --> 00:03:07.599
um not a ter like crazy large amount of it.

s30
00:03:07.599 --> 00:03:10.800
And then these black boxes are high bandwidth memory.

s31
00:03:10.800 --> 00:03:19.680
So when you Nvidia Smi and see you know 80 gigabytes, 1008 gigabytes, whatever it is on your GPU, that's that memory.

s32
00:03:19.680 --> 00:03:24.159
Um a GPU is operating a highly parallel program.

s33
00:03:24.159 --> 00:03:30.640
So multiple threads are combined together in uh into larger coarser units.

s34
00:03:30.640 --> 00:03:38.640
and we schedule these threads and and blocks onto these processors to perform our AI compute.

s35
00:03:38.640 --> 00:03:47.280
Beyond the GPU, we'll have multiple GPUs and we'll also have, you know, CPUs um that have memory as well.

s36
00:03:49.040 --> 00:04:01.360
Um so to perform computation um the memory that these threads use is going to be stored in a really fast register memory that's right next to the computation units.

s37
00:04:01.360 --> 00:04:17.840
um simple physics if I am pulling data from very very close to my proc my compute unit it's really fast to get to it because it you know that data is right next to me but there's not a large radius and not a large volume of space that's close by to my

s38
00:04:17.840 --> 00:04:29.280
process uh my my compute units and so I don't have very much of it so you can see that the fastest memory here the registers is sup is you know 130 terabytes per second on an H100

s39
00:04:29.280 --> 00:04:37.680
but we don't have very much on of it and as we go to the further away memory we have a lot more of it but it takes longer to reach it.

s40
00:04:37.680 --> 00:04:40.720
Uh again simple physics.

s41
00:04:40.720 --> 00:04:46.720
So in multiGPU systems in particular um there is a hierarchy of interconnects.

s42
00:04:46.720 --> 00:04:53.360
So we will have something called PCIe um as the channel for CPUGGPU communications.

s43
00:04:53.360 --> 00:05:04.080
We'll have multiGPU or multi- uh node communications over infiniband TCP and then in the we're going to focus mostly on the Nvidia sphere here.

s44
00:05:04.080 --> 00:05:14.479
Um in uh the intraGPU regime we'll have NVLink um providing point-to-point connections between GPUs and the NV switch.

s45
00:05:14.479 --> 00:05:29.759
NV switch connects all NVLink in endpoints into a non-blocking fabric for full GPUGGPU communication and NV switch is exciting because it also provides support for in network offdevice acceleration

s46
00:05:29.759 --> 00:05:36.560
for oper like communication primitives like multiccast and reductions.

s47
00:05:37.680 --> 00:05:41.520
Okay, so diving in with the preliminaries in mind.

s48
00:05:41.520 --> 00:05:44.960
Why GPU networking now?

s49
00:05:44.960 --> 00:05:53.039
So as I mentioned at the beginning, we've really put a lot of effort into making AIO uh more efficient over recent years.

s50
00:05:53.039 --> 00:06:00.720
Um again, we have uh architectures that use less compute, less memory like Mamba or sparse attentions.

s51
00:06:00.720 --> 00:06:06.400
We have algorithms that make AI more hardware affair aware like flash attention.

s52
00:06:06.400 --> 00:06:15.039
We have tools to make it easy to map AI algorithms to the hardware like tileang, mojo, triton, gluon, thunder kittens.

s53
00:06:15.039 --> 00:06:23.840
Um, and we have new techniques to overlap uh execution across many AI operators very tightly like mega kernels.

s54
00:06:23.840 --> 00:06:35.520
And we also have tools to make it easy to run on multiple vendor and silicon platforms like thunder mittens for Apple silicon or or hipkittens for AMD and so on.

s55
00:06:35.520 --> 00:06:44.800
Um, at this point we really believe that GPU networking offers many new and exciting opportunities for AI efficiency.

s56
00:06:45.360 --> 00:06:52.240
Modern AI workloads are getting very big and require kernels that span multiple um, you know, GPUs.

s57
00:06:52.240 --> 00:07:07.440
So on many production um distributed training and inference workloads, communication is increasingly consuming the majority of the runtime and yields low model flop utilization at scale.

s58
00:07:08.560 --> 00:07:19.840
So the uh pace of innovation and diversity of approaches that different hardware providers are taking in their networking stacks is another reason why it's an exciting time to study

s59
00:07:19.840 --> 00:07:22.479
um you know networking and communication.

s60
00:07:22.479 --> 00:07:35.199
So we can see here um AMD hardware with uh what's called XGMI interconnects um providing pointto-point links between different GPUs in a scaleup domain.

s61
00:07:35.199 --> 00:07:39.840
We can see here TPU and uh interconnect uh as well.

s62
00:07:39.840 --> 00:07:45.280
So the TPU will use a 3D Taurus and also have optical wraparound links.

s63
00:07:45.280 --> 00:07:49.840
So yet another diverse form of the topology and links.

s64
00:07:49.840 --> 00:08:07.759
Um and then again for Nvidia we'll have the INV um switch which integrates um compute capabilities directly into the interconnect fabric like for in network reductions and then we'll have our NV link providing up to you know 900 gigabytes of unidirectional

s65
00:08:07.759 --> 00:08:24.080
bandwidth between any two remote GPUs high bandwidth memory um on you know particular generation of Nvidia hardware um Beyond the diversity in the networking stacks, there's also a lot of evolution

s66
00:08:24.080 --> 00:08:37.919
in um how AI workloads are adapting to take advantage of this hardware and the diversity of types of um you know hardware that we're we're using simultaneously for one AI workload.

s67
00:08:37.919 --> 00:08:46.240
So um KV cache memory in modern inference systems is going to beworked across GPU, CPU, disk and remote machines.

s68
00:08:46.240 --> 00:08:53.440
um inference systems increasingly disagregate different steps of inference across different hardware backends.

s69
00:08:53.440 --> 00:09:00.240
So you could run speculative decoding on some hardware, decode on different hardware, prefill on different hardware.

s70
00:09:00.240 --> 00:09:13.920
Um hardware is also evolving to have larger scaleup domains than ever before with you know 72 GPUs and a scaleup domain in in the coming um chips and Nvidia planning on a single system in 2027

s71
00:09:13.920 --> 00:09:17.120
with 576 GPUs.

s72
00:09:17.120 --> 00:09:28.640
Um at the same time to take advantage of these more uh intensive scaleup domains we're getting richer uh primitives for fine grained control and kernel writing over these domains.

s73
00:09:28.640 --> 00:09:40.160
So we have something called tensor memory acceleration where we can provide uh perform asynchronous network transfers from the device side um on these GPUs.

s74
00:09:40.160 --> 00:09:53.120
So all of these changes are opening up new opportunities and challenges in um both AI you know how do we build models that take advantage of these trends and in systems.

s75
00:09:53.839 --> 00:10:06.080
Okay, so the problems that we're going to go after um how do we get peak hardware utilization and also development simplicity um for these multiGPU kernels.

s76
00:10:06.640 --> 00:10:17.760
So um it's been very difficult to write multiGGPU kernels and there's a lot of there are a lot of papers a lot of systems reports that document you know challenges here.

s77
00:10:17.760 --> 00:10:29.440
Um, one of the things here is it's compounded by the fact that communication hardware around GPUs has progressed a lot more slowly relative to compute and memory.

s78
00:10:29.440 --> 00:10:40.880
Um, so comparing NVIDIA A100's in 2020 to B200s in 2024, u BF16 tensor core speeds improved by 7.2x.

s79
00:10:40.880 --> 00:10:47.760
um while intra node communication by just 3x and inter node communication by just 2x.

s80
00:10:47.760 --> 00:11:02.800
Um and coming back to my points about how diverse networking is right now things like tensor cores that run map moles and our memory hierarchies are pretty consistent and resemble one another across diverse AI vendors and multisilicon.

s81
00:11:02.800 --> 00:11:15.279
Um but again as I I mentioned the networking stack is something that is really different across vendors still um you know a first step as we went about all this work is to just study the baselines

s82
00:11:15.279 --> 00:11:16.399
um that are out there.

s83
00:11:16.399 --> 00:11:31.839
So um one of the popular tools for um communications is this nickel library or Rickle on AMD um that both you know companies respectively spend a lot of you know engineering investment into

s84
00:11:31.839 --> 00:11:35.519
releasing to make it easy for people to do multiGPU work.

s85
00:11:35.519 --> 00:11:37.519
Um but they're not very flexible.

s86
00:11:37.519 --> 00:11:43.200
So they're tuned for bulk transfers for large contiguous chunks of data transfers.

s87
00:11:43.200 --> 00:11:54.160
And the design really breaks down when you care about peak performance, fine grain communication, um, and sort of non-trivial collectives that you want to fuse together.

s88
00:11:54.160 --> 00:12:03.200
So as a result, you can achieve much higher performance by writing custom communication kernels that directly address these needs.

s89
00:12:03.200 --> 00:12:16.480
If we look at a naive baseline that's representative of very popular libraries in machine learning stacking pietorch with nickel um we can we find that across um you know the many

s90
00:12:16.480 --> 00:12:30.079
uh problems in our parallel kernel bench benchmark that the majority of these um simple baselines will fall below 50% of their communication aware roof line bound.

s91
00:12:30.079 --> 00:12:33.760
So there's a lot of room for improvement here.

s92
00:12:33.760 --> 00:12:48.079
Um the current frameworks beyond um nickel which is popular in like Megatron LM, Flex Flow, Nanoflow um you know all again these systems are primarily orchestrating bulk collectives via nickel

s93
00:12:48.079 --> 00:12:52.240
um and require synchronization before and after data transfers.

s94
00:12:52.240 --> 00:12:58.560
So beyond th those off-the-shelf libraries, we have compilers and DSLs that exist.

s95
00:12:58.560 --> 00:13:01.920
So there's Triton distributed is one of them.

s96
00:13:01.920 --> 00:13:05.680
Um and uh you know tile link is another one.

s97
00:13:05.680 --> 00:13:13.680
Um we have found that it's very difficult to support the rapid pace of networking improvements within these frameworks.

s98
00:13:13.680 --> 00:13:27.760
So our benchmarks and our papers highlight results where Triton distributed originally tuned around 8 uh H800 GPUs fails to adapt efficiently to other architectures like H100s.

s99
00:13:27.920 --> 00:13:36.720
And then the third category of how people can proceed here is to really handtune specific AI operators one by one.

s100
00:13:36.720 --> 00:13:52.639
So there's a lot of popular work um DPP um comet ring attention um flux flashdoe um and then several distributed gem kernels from cutless and these methods achieve peak performance

s101
00:13:52.639 --> 00:14:02.399
but often um they do not you know some of these methods have been designed in one precision and it takes five or six months to scale it to another precision

s102
00:14:02.399 --> 00:14:10.560
and just the scalability of this hand tuning and um fine grain kernel writing is not very um effective.

s103
00:14:10.639 --> 00:14:27.600
So um with this landscape in mind, our research question was really about whether there is a small set of principles and fundamentals that really governs multiGPU kernel writing and whether there are methods that can leverage those principles if they exist

s104
00:14:27.600 --> 00:14:31.600
to simplify the development of these kernels.

s105
00:14:32.000 --> 00:14:38.880
Um I'll briefly highlight two works here that um govern like that that represent our approach.

s106
00:14:38.880 --> 00:14:49.920
So first we think it's important to build our own fundamental understanding and to manually do the work to understand it rather than just throwing say an LLM at the problem.

s107
00:14:49.920 --> 00:14:58.959
So we spent the time to build out parallel kittens which is a small set of minimal primitives and patterns for multiGPU kernels.

s108
00:14:58.959 --> 00:15:11.680
Um we use this to understand the trade-offs of multiGPU kernels and to one write a large collection of um peak performance kernels for a variety of parallelism schemes.

s109
00:15:11.680 --> 00:15:18.880
Um and this I will use to hopefully you know educate and bring us all on the same page on what patterns we figured out.

s110
00:15:18.880 --> 00:15:33.279
Um and then once we found that there is indeed a small set of trade-offs governing this landscape, we were curious whether models, especially these models right now that claim to be very good at kernel writing and also reasoning

s111
00:15:33.279 --> 00:15:42.959
um could reason about these trade-offs when we provide them in context to actually generate a bunch of net new multiGPU kernels for us.

s112
00:15:42.959 --> 00:15:48.000
Unfortunately, we found they were not very good, but we'll dive into more of that at the end.

s113
00:15:48.000 --> 00:15:56.320
Um so just the fundamental section this is going to be more you know educational what are the trade-offs that go into these kernels.

s114
00:15:56.320 --> 00:16:01.920
So there are three main ways to do um intraGPU data transfers.

s115
00:16:01.920 --> 00:16:09.360
Um there's the per GPU what's called copy engine and this is host or CPU initiated work.

s116
00:16:09.360 --> 00:16:12.480
It's really good for large message transfers.

s117
00:16:12.480 --> 00:16:23.519
So when your message size, the amount of data being transferred is really big um and it can get to sort of like peak bandwidth um on on the communication side.

s118
00:16:23.519 --> 00:16:37.759
In contrast, you can use device initiated or GPU initiated transfers via that tensor memory accelerator that I mentioned or via register level instructions called uh in sort of their PTX lingo like LDST

s119
00:16:37.759 --> 00:16:39.360
red multime.

s120
00:16:39.360 --> 00:16:43.199
Um, and the uh T TMA is really nice.

s121
00:16:43.199 --> 00:16:52.240
These device initiated ones are really nice because they can saturate our NVLink bandwidth using relatively small message sizes.

s122
00:16:52.240 --> 00:17:02.480
And this means that they can be really nice when we're trying to do fine grain communication rather than sending bulk amounts of data over the links all at once coarsely.

s123
00:17:02.639 --> 00:17:04.799
Um, there are some trade-offs here.

s124
00:17:04.799 --> 00:17:15.919
So the copy engine is really nice because it doesn't take away or you know waste a lot of our precious registers that I mentioned are important for compute on the GPU.

s125
00:17:15.919 --> 00:17:29.679
Um and it doesn't also use any of those rainbow colored dots the processors on our GPU allowing us to repurpose those for memory or um computation on our uh you know other parts of the AI pipeline.

s126
00:17:29.679 --> 00:17:35.760
Um, TMA, the second option here, consumes very few registers, which is why it's nice.

s127
00:17:35.760 --> 00:17:41.440
Um, and it also can achieve high utilization using very few of our processors.

s128
00:17:41.440 --> 00:17:45.840
So, it's a nice useful tool for fine grain overlapping.

s129
00:17:45.840 --> 00:17:48.000
Um, TMA does have limitations.

s130
00:17:48.000 --> 00:17:56.559
It can't effectively take advantage of these in network um, computations that I mentioned are feasible with technologies like NV switch.

s131
00:17:56.559 --> 00:18:08.880
And the register level instructions are really nice for being able to take advantage of um you know those those sort of in network reductions that NV switch offers.

s132
00:18:08.880 --> 00:18:17.919
So again there are different tradeoffs different functionalities that these transfer mechanisms offer and they face different trade-offs.

s133
00:18:18.720 --> 00:18:29.600
Second um beyond transfer mechanism the trade-off is around how to overlap compute memory and uh communication in GPU kernels.

s134
00:18:29.600 --> 00:18:33.280
So there's two main categories of schedules.

s135
00:18:33.280 --> 00:18:47.360
The first is intraSM within one of those rainbow dots um where we'll we'll have different warps or threads within that processor specialized to handle either compute or one specialized for communication

s136
00:18:47.360 --> 00:18:48.640
concurrently.

s137
00:18:48.640 --> 00:18:52.160
Um we can dedicate you know different warps to each of these.

s138
00:18:52.160 --> 00:19:01.200
The challenge with this intm overlapping is that the communication and computation pattern really need to like align and jive with one another.

s139
00:19:01.200 --> 00:19:06.880
they need to use the same data as inputs for the computation and communication.

s140
00:19:06.880 --> 00:19:21.360
When they don't align, you could use something like interm um schedules that are shown on your right here where we'll now have each of the different rainbow colored dots on our GPUs, those different processors specialized to

s141
00:19:21.360 --> 00:19:25.360
compute communication and memory.

s142
00:19:25.360 --> 00:19:38.799
Um and so this this is nice when the colonel would others wise need to split across resources like the register file or shared memory um across these different steps in misaligned

s143
00:19:38.799 --> 00:19:40.799
ways.

s144
00:19:41.200 --> 00:19:49.520
Um this is also really nice when it's hard to maximize NVLink traversal with intram overlapping.

s145
00:19:50.080 --> 00:20:00.080
So I just wanted to highlight one quick example here where each of the patterns excels in popular you know AI uh uh like kind of patterns that you'll see.

s146
00:20:00.080 --> 00:20:16.000
So on a gem plus uh reduced uh scatter here we can see that the um intraSM overlapping scheduleuler schedule is very effective in the gem plus all reduce we can see that the interSM

s147
00:20:16.000 --> 00:20:21.039
which again leverages the in network reductions of envy switch is very effective.

s148
00:20:21.039 --> 00:20:30.159
So we face these trade-offs and you can read more about um the design decisions that go into them in our parallel kittens paper.

s149
00:20:30.159 --> 00:20:41.760
Um, and then finally, ideally, abstraction should allow the developer flexibility to control how they're buffering and synchronizing between data senders and receivers.

s150
00:20:41.760 --> 00:20:45.600
So, we encapsulated these ideas into parallel kittens.

s151
00:20:45.600 --> 00:20:51.280
Again, a simple set of programming primitives and templates for these multiGPU kernels.

s152
00:20:51.280 --> 00:21:01.039
Parallel Kittens is used in production at Together AI as well as our partner um you know Cursor and other uh companies in the AI space.

s153
00:21:01.039 --> 00:21:02.880
Um here's some sample code.

s154
00:21:02.880 --> 00:21:13.039
I won't spend too much time here, but we usually add roughly a dozen lines of code over a single GPU kernel to insert these multiGPU primitives.

s155
00:21:13.840 --> 00:21:26.400
Um, and you can see here across data sequence and expert parallelism how our parallel kittens kernels are achieving state-of-the-art results um, compared to strong reference baselines.

s156
00:21:26.400 --> 00:21:30.480
And you can check out our repo to learn more.

s157
00:21:30.799 --> 00:21:40.320
Okay, so we understand a little bit about um the trade-offs that underly these multiGPU kernels and there's just a couple main, you know, ones that exist.

s158
00:21:40.320 --> 00:21:48.320
So can models reason through them um and give us these uh you know kernels models are getting better at reasoning today.

s159
00:21:48.320 --> 00:22:02.559
Um do they generalize well to these problems or are we benchmaxed on you know benchmarks of the past which are more single GPU centric uh models right now are showing really promising results on single GPU

s160
00:22:02.559 --> 00:22:03.679
uh benchmarks.

s161
00:22:03.679 --> 00:22:07.360
So it's a ripe time to to extend it.

s162
00:22:07.360 --> 00:22:25.840
In our benchmark parallel kernel bench, each task presents the model with an unoptimized reference implementation written in PyTorch with torch distributed um nickel operations and then a system topology that specifies the number of ranks and intraode hardware configuration.

s163
00:22:25.840 --> 00:22:34.720
And the model needs to rewrite the reference into a performance CUDA kernel that uh uses unified virtual addressing.

s164
00:22:37.360 --> 00:22:44.960
The um multiGPU problem space expands combinatorally beyond single GPU um cases.

s165
00:22:44.960 --> 00:22:53.280
So a standard transformer layer can be parallelized across data sequence, tensor, context, layer, pipeline and expert dimensions.

s166
00:22:53.280 --> 00:22:58.320
and each composition um induces a different communication pass uh pattern.

s167
00:22:58.320 --> 00:23:11.679
So to make sure that our benchmark has high coverage over the representative types of multiGPU problems um we created this taxonomy um that you can read more about in our paper and then picked representative

s168
00:23:11.679 --> 00:23:15.039
problems for each part of the taxonomy.

s169
00:23:15.280 --> 00:23:23.039
Um these are all patterns that arise in real AI workloads from inference to RL to post-training.

s170
00:23:23.440 --> 00:23:31.360
There are 87 problems overall um drawn from GitHub repositories that we found to be very informative.

s171
00:23:31.360 --> 00:23:40.799
Um and we uh and like optimized library implementations and DSLs that people have written multiGPU kernels in.

s172
00:23:40.799 --> 00:23:51.919
Um, we wanted to really make sure that solving PK this parallel kernel bench would lead to net new useful production kernels rather than artificial or useless kernels.

s173
00:23:51.919 --> 00:23:54.400
Okay, so that's parallel kernel bench.

s174
00:23:54.400 --> 00:23:57.200
How do models perform?

s175
00:23:58.799 --> 00:24:01.440
So we measured to sorry that this is a bit small.

s176
00:24:01.440 --> 00:24:03.200
We measured two main metrics.

s177
00:24:03.200 --> 00:24:17.360
Um pass at K which is the number of correct kernels generated after K attempts and then fast um one at K which counts solutions that are both correct and outperform

s178
00:24:17.360 --> 00:24:21.600
the speed of the pi torch plus nickel baseline.

s179
00:24:21.600 --> 00:24:29.919
Um so pass K just correctness fast one at K is whether you're getting a 1x or higher speed up over the reference.

s180
00:24:29.919 --> 00:24:43.600
So performanceoriented um we found that in the zeroot setting the best of the frontier models we tried solves 28 out of 87 problems and 22 of those problems are faster than the pietorch

s181
00:24:43.600 --> 00:24:45.840
plus nickel baseline.

s182
00:24:45.840 --> 00:24:59.200
If we make multiple samples, you know, standard uh scaling up test time compute, we can uh get that number from say like to to 36 correct solutions, but the fast

s183
00:24:59.200 --> 00:25:04.159
uh one performance still plateaus out at roughly 31%.

s184
00:25:04.159 --> 00:25:12.159
So we don't see much room from continuing to scale there as we increase the number of parallel generations.

s185
00:25:12.320 --> 00:25:24.000
Um we find that the correct once correctness is established um speedups naturally come from eliminating nickel staging overhead in favor of direct NVLink loads in stores.

s186
00:25:24.240 --> 00:25:30.000
Um the success patterns here are really concentrated into familiar patterns.

s187
00:25:30.000 --> 00:25:35.520
So collective primitives, tensor parallel gems and Ulyses style context parallelism.

s188
00:25:35.520 --> 00:25:49.279
So in other words, patterns that we see heavily represented um on the internet rather than necessarily patterns that the model has used its reasoning abilities to think through.

s189
00:25:50.880 --> 00:26:00.400
Okay, even the best available model that we benchmarked here GPT 5.5 drops off very quickly as the speed up threshold increases.

s190
00:26:00.400 --> 00:26:07.200
So on the x-axis here, we're increasing the speed up threshold over that pietor torch plus nickel baseline.

s191
00:26:07.200 --> 00:26:16.880
And then we're showing the number of correct kernels that are faster than that baseline or this much faster than the baseline on the y-axis.

s192
00:26:16.880 --> 00:26:26.400
So GPT 5.5 is this orange line here and then DeepSeek V4 Pro is the aqua line at the bottom.

s193
00:26:28.720 --> 00:26:32.880
um we found that there's deeper issues than CUDA syntax.

s194
00:26:32.880 --> 00:26:42.640
So we found that if you do multiple sampling or have the model kind of look at its errors and correct them, it can often compile the kernels.

s195
00:26:42.640 --> 00:26:48.799
But the models really struggle to reason through the tradeoffs that we talked about in the prior section.

s196
00:26:48.799 --> 00:26:57.440
um collective ordering, data partitioning, thinking about intra versus interm scheduling or deciding between the different transfer mechanisms.

s197
00:26:57.440 --> 00:27:07.200
Um we find that they often do not use things like the register transfer instructions or tensor memory acceleration when writing the kernels.

s198
00:27:07.760 --> 00:27:14.720
Um we wanted to try a pretty simple instantiation of something like a clawed code um coding agent.

s199
00:27:14.720 --> 00:27:29.039
So, we took uh the mini sui agent multi-turn harness and one of the best performing models, Gemini 3 Pro, and gave it access to a local bash environment to sort of mimic the standard claude code setup.

s200
00:27:29.039 --> 00:27:44.240
We found that this could help the agent um go from solving 24 problems to 35 of 87 problems um with um 26 achieving over a 1x speed up over the reference.

s201
00:27:44.240 --> 00:27:56.240
But we found that as we scaled the amount of time um the performance plateaued as uh and we find that additional techniques would be required to continue seeing the scaling there.

s202
00:27:56.240 --> 00:28:00.080
Again these results are discussed in more detail in our paper.

s203
00:28:00.080 --> 00:28:08.559
Um we think this is a really exciting you know just to wrap up here we hope that people out here can use both parallel kittens and parallel kernel bench.

s204
00:28:08.559 --> 00:28:19.200
We think that the kernels generated from solving parallel kernel bench will lead to net new production kernels that are important bottlenecks for inference in RL right now.

s205
00:28:19.200 --> 00:28:30.240
um they're you know we tried our best to make them you know non-artificial and we can already see signs of life and exciting results where people have not invested a bunch of time to handw write a multiGPU

s206
00:28:30.240 --> 00:28:48.000
kernel and we've gotten some net new interesting ones like this Nemo vocab parallel um you know filtering kernel um a hyena architecture context parallelism kernel and uh the SAM 3 video segmentation

s207
00:28:48.000 --> 00:28:52.240
model um IOU suppression kernel.

s208
00:28:52.320 --> 00:29:00.240
Just to conclude here um we're really excited about um how uh well just talking about the lessons.

s209
00:29:00.240 --> 00:29:11.200
First off, we think there aren't that many patterns that are involved in writing intragpu effective kernels again encapsulated by our small set of programming primitives.

s210
00:29:11.200 --> 00:29:18.640
Um, but unfortunately models do not currently understand how to reason through these trade-offs even when we provide them in context.

s211
00:29:18.640 --> 00:29:22.880
Um, we're really excited about methods that can help attack this benchmark.

s212
00:29:22.880 --> 00:29:29.039
We're excited about architectures that can grow with the trends of how networking stacks are evolving.

s213
00:29:29.039 --> 00:29:36.159
Um, you know, larger scale up domains shift away from scale out um and massive onchip memory structures.

s214
00:29:36.159 --> 00:29:42.480
And we hope that uh you know we can also extend and you you can feel free to reach out to me at my email.

s215
00:29:42.480 --> 00:29:44.480
Thanks.