{ "video": { "id": "pOvWgX7IJsc", "title": "Can LLMs Write Fast Multi-GPU Kernels? — Simran Arora, Together AI", "duration": 1800, "upload_date": null, "channel": "AI Engineer", "source": "AI Engineer" }, "analysis": { "video_id": "pOvWgX7IJsc", "title": "Can LLMs Write Fast Multi-GPU Kernels? — Simran Arora, Together AI", "one_liner": "Together AI's Simran Arora argues the AI performance bottleneck has moved from single-GPU kernels to multi-GPU communication, and shows on their new 87-problem ParallelKernelBench that frontier models — best case 28/87 zero-shot — can compile CUDA but can't reason through the handful of trade-offs that actually govern fast multi-GPU kernels.", "summary": "After years of investment in flash attention, memory-efficient architectures and single-GPU DSLs, the bottleneck has shifted to GPU networking, where communication now eats the majority of runtime in production distributed training and inference. Arora's team built ParallelKittens, a minimal set of primitives capturing the small number of real trade-offs (transfer mechanism: copy engine vs TMA vs register-level multimem instructions; schedule: intra-SM vs inter-SM overlap; buffering/synchronization control), and used it to write state-of-the-art kernels across data, sequence and expert parallelism. They then asked whether frontier models can apply those same principles when handed them in context, via ParallelKernelBench — 87 problems drawn from real GitHub repos and library implementations. The answer is no: models succeed mostly on patterns heavily represented on the internet, and performance plateaus as you scale samples or agent time.", "key_points": [ "Communication hardware has fallen behind compute: from NVIDIA A100 (2020) to B200 (2024), BF16 tensor core speed improved 7.2x while intra-node communication improved just 3x and inter-node just 2x.", "A naive PyTorch + NCCL baseline falls below 50% of its communication-aware roofline bound on the majority of ParallelKernelBench problems; NCCL/RCCL are tuned for bulk contiguous transfers and break down for fine-grained communication and fused non-trivial collectives.", "Existing DSLs don't keep up with networking churn — Triton-distributed, originally tuned around 8 H800 GPUs, fails to adapt efficiently to H100s — and hand-tuning operators one by one (DPP, Comet, ring attention, Flux, FlashDMoE, CUTLASS distributed GEMMs) hits peak performance but can take five or six months just to port to another precision.", "The trade-off space is small and concrete: copy engine (host-initiated, best for large messages, burns no registers or SMs); TMA (device-initiated, saturates NVLink with small messages, few registers, few SMs, but can't exploit NVSwitch in-network compute); register-level PTX instructions like LD/ST-reduce-multimem (can exploit NVSwitch in-network reductions). Schedules split into intra-SM warp specialization (needs compute and comms to share input data) vs inter-SM specialization.", "Empirically the schedules split by workload: GEMM + reduce-scatter favours intra-SM overlap; GEMM + all-reduce favours inter-SM, which leverages NVSwitch's in-network reductions.", "ParallelKittens encodes these into templates adding roughly a dozen lines over a single-GPU kernel; it is in production at Together AI and at Cursor, and hits state-of-the-art across data, sequence and expert parallelism.", "ParallelKernelBench: 87 problems, each giving the model an unoptimized PyTorch + torch.distributed/NCCL reference plus a system topology (rank count, intra-node hardware config), asking for a performant CUDA kernel using unified virtual addressing; scored with pass@k and fast_1@k (correct AND faster than the PyTorch+NCCL baseline).", "Results: best frontier model solves 28/87 zero-shot with 22 beating the baseline; scaling parallel samples reaches ~36 correct but fast_1 plateaus around 31%. GPT-5.5 leads and DeepSeek V4 Pro trails, and GPT-5.5's count drops off fast as the required speedup threshold rises. A mini-SWE-agent harness with Gemini 3 Pro and a local bash environment (a stand-in for a Claude Code setup) went from 24 to 35 of 87 solved with 26 above 1x, then also plateaued with more time.", "Failures are not CUDA syntax — with retries models compile fine; they fail at collective ordering, data partitioning, intra- vs inter-SM scheduling and choosing transfer mechanisms, and often never reach for register-level transfer instructions or TMA. Successes cluster on collective primitives, tensor-parallel GEMMs and Ulysses-style context parallelism.", "Solving the benchmark already yields net-new production kernels nobody had hand-written: a Nemo vocab-parallel filtering kernel, a Hyena-architecture context-parallelism kernel, and an IOU suppression kernel for the SAM 3 video segmentation model." ], "takeaways": [ "Stop treating PyTorch + NCCL as a performance floor worth accepting — measure against a communication-aware roofline, and expect custom kernels using direct NVLink loads/stores to win largely by eliminating NCCL's staging overhead.", "Choose the transfer mechanism deliberately rather than by default: copy engine for bulk messages when you want to keep registers and SMs free, TMA for fine-grained device-initiated transfers, and register-level multimem instructions when you want NVSwitch's in-network reductions.", "Pick the overlap schedule from the operator shape — intra-SM warp specialization when compute and communication consume the same data, inter-SM when they'd otherwise fight over the register file or shared memory, or when intra-SM can't saturate NVLink.", "Don't outsource multi-GPU kernel design to an LLM yet, and don't assume more samples or more agent turns will fix it — both plateau; build the fundamental understanding first, then use the model within primitives like ParallelKittens.", "Design for where the hardware is going: scale-up domains of 72 GPUs today and an announced 576-GPU single system from NVIDIA in 2027, with KV cache tiered across GPU, CPU, disk and remote machines and inference stages disaggregated across different backends." ], "topics": [ "gpu-kernels", "multi-gpu", "benchmarks", "code-generation", "distributed-training", "inference-optimization", "cuda", "llm-reasoning" ], "tools": [ "Together AI", "ParallelKittens", "ParallelKernelBench", "ThunderKittens", "ThunderMittens", "HipKittens", "NCCL", "RCCL", "PyTorch", "torch.distributed", "Triton", "Triton-distributed", "TileLang", "TileLink", "Mojo", "Gluon", "CUTLASS", "Megatron-LM", "FlexFlow", "NanoFlow", "FlashAttention", "Mamba", "NVIDIA H100", "NVIDIA A100", "NVIDIA B200", "H800", "NVLink", "NVSwitch", "PCIe", "InfiniBand", "AMD XGMI", "TPU", "GPT-5.5", "DeepSeek V4 Pro", "Gemini 3 Pro", "mini-SWE-agent", "Claude Code", "Cursor", "Nemo", "SAM 3", "Hyena", "DeepSeek", "Stanford Hazy Research", "Caltech" ], "quotes": [ { "text": "comparing NVIDIA A100's in 2020 to B200s in 2024, BF16 tensor core speeds improved by 7.2x, while intra node communication by just 3x and inter node communication by just 2x.", "at": "10:30", "url": "https://www.youtube.com/watch?v=pOvWgX7IJsc&t=630s" }, { "text": "we think it's important to build our own fundamental understanding and to manually do the work to understand it rather than just throwing say an LLM at the problem.", "at": "14:37", "url": "https://www.youtube.com/watch?v=pOvWgX7IJsc&t=877s" }, { "text": "So in other words, patterns that we see heavily represented on the internet rather than necessarily patterns that the model has used its reasoning abilities to think through.", "at": "25:37", "url": "https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1537s" }, { "text": "we think there aren't that many patterns that are involved in writing intragpu effective kernels... but unfortunately models do not currently understand how to reason through these trade-offs even when we provide them in context.", "at": "29:00", "url": "https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1740s" } ], "words": 4849 }, "summary_url": "/#pOvWgX7IJsc", "transcript": { "html": "/transcripts/pOvWgX7IJsc.html", "txt": "/transcripts/pOvWgX7IJsc.txt", "vtt": "/transcripts/pOvWgX7IJsc.vtt" } }