{ "video": { "id": "7BcQdtRA3Tc", "title": "Taming the AI Hardware Cambrian Explosion — Abdul Dakkak, Modular", "duration": 1028, "upload_date": null, "channel": "AI Engineer", "source": "AI Engineer", "event_format": "" }, "analysis": { "video_id": "7BcQdtRA3Tc", "title": "Taming the AI Hardware Cambrian Explosion — Abdul Dakkak, Modular", "one_liner": "Modular says one vertically integrated stack can tame today's hardware 'Cambrian explosion': Mojo, a Pythonic kernel language, under MAX, an inference engine built to replace the PyTorch-plus-vLLM/SGLang layers, under Modular Cloud. It reports independently verified speedups such as 5.5x faster TTFT than vLLM on Kimi K2.5 and about 3x faster than the next competitor on Flux 2 dev.", "summary": "Abdul Dakkak argues that the compute crunch has produced an explosion of hardware (NVIDIA, AMD, Intel, ARM, Apple) and a messy software layer where hardware choices 'bubble up' into vLLM, SGLang, TRT-LLM and llama.cpp. Modular rebuilt the stack from first principles in three layers. Mojo is a Pythonic systems language for CPUs, GPUs and accelerators. MAX is a portable inference engine and framework built on Mojo kernels. Modular Cloud is a 'glass box' platform for distributed, heterogeneous serving. He backs this with benchmarks against the best vLLM/SGLang setups and with Artificial Analysis results, many of them described as independently verified.", "key_points": [ "Positioning: CUDA, ROCm and similar give peak performance but sacrifice usability and portability. OpenCL gave portability but sacrificed performance and usability. Triton gives about 70% of the performance with portability and usability. Mojo aims for both: portable, usable kernels plus low-level access for maximum performance.", "Modular has open-sourced practically all its kernels, over 700,000 lines of kernel code, plus model definitions and MAX Serve. It claims its matmul beats cuBLAS and hipBLAS, and its attention kernel is faster than FlashAttention.", "Mojo 1.0, which stabilises the APIs, was announced about a month earlier and will ship in the next few months. Mojo itself will also be open-sourced in the next few months.", "MAX is pitched as a fix for PyTorch's assumptions. PyTorch is 'about 10 years old' and not designed for GenAI inference, which is why vLLM and SGLang stepped in. MAX has graph compilers, eager mode for debugging, Python model authoring and fusion across Mojo programs. It supports thousands of architectures (e.g. Gemma 4, MiniMax, GLM), exposes OpenAI-compatible endpoints, and handles text, image, video and audio in one framework.", "MAX APIs natively program multi-GPU and multi-node systems, and these can be heterogeneous, for example prefill on hardware X and decode on hardware Y.", "Modular Cloud offers dedicated endpoints, bring-your-own-cloud and on-prem deployment, custom or fine-tuned models with custom speculative decoders, and a Cloud Lab for tuning parameters to choose a point on the Pareto frontier for a given workload.", "Benchmarks: Kimi K2.5 is 5.5x faster TTFT than vLLM, with 1.5x tokens per minute per GPU on B200. Flux 2 is about 7x faster than torch.compile with diffusers. On AMD MI355 it is 4x faster than the equivalent, and AMD is usually about 30% cheaper per GPU than the NVIDIA equivalent, so TCO matters. Artificial Analysis shows about 3x faster than the next competitor on Flux 2 dev with no distillation or quantization tricks. On Gemma 4 it is about 6x faster than SambaNova.", "Fast inference enables new applications: sub-second image generation (about 0.3s in many cases), and a booth demo that reads a live video feed and re-renders it from a prompt." ], "takeaways": [ "Avoid locking inference onto one hardware vendor. Compare cost per GPU (e.g. AMD about 30% cheaper) alongside raw speedups when judging TCO.", "Benchmark Modular yourself: run `pip install modular`, then check the open-sourced kernels and the Artificial Analysis results against your current vLLM or SGLang setup.", "Consider disaggregated, heterogeneous serving, such as prefill and decode on different hardware, and tune deployments along the cost and latency Pareto frontier for your workload.", "If you're new to Mojo or MAX, use Modular's AI skills to port existing code into Mojo or MAX." ], "topics": [ "ai-hardware", "inference", "gpu-kernels", "programming-languages", "model-serving", "distributed-inference", "benchmarks", "portability" ], "tools": [ "Modular", "Mojo", "MAX", "Modular Cloud", "Modular Cloud Lab", "NVIDIA", "AMD", "Intel", "ARM", "Apple", "vLLM", "SGLang", "TensorRT-LLM", "llama.cpp", "PyTorch", "CUDA", "ROCm", "OpenCL", "Triton", "cuBLAS", "hipBLAS", "FlashAttention", "Kimi K2.5", "Flux 2", "Gemma 4", "MiniMax", "GLM", "diffusers", "torch.compile", "B200", "MI355", "Artificial Analysis", "SambaNova", "OpenAI-compatible API", "Kubernetes" ], "quotes": [ { "text": "So folks that actually know Python today pretty much know 90% of Mojo.", "at": "02:23", "url": "https://www.youtube.com/watch?v=7BcQdtRA3Tc&t=143s" }, { "text": "PyTorch is about 10 years old. It was not really designed for the genai inference of today.", "at": "06:29", "url": "https://www.youtube.com/watch?v=7BcQdtRA3Tc&t=389s" }, { "text": "there's nothing to stop you from saying the prefill is on hardware X and the decode is on hardware Y.", "at": "09:20", "url": "https://www.youtube.com/watch?v=7BcQdtRA3Tc&t=560s" }, { "text": "This is just raw speed because we can leverage the hardware from the ground up", "at": "13:46", "url": "https://www.youtube.com/watch?v=7BcQdtRA3Tc&t=826s" } ], "words": 2891 }, "summary_url": "/#7BcQdtRA3Tc", "transcript": { "html": "/transcripts/7BcQdtRA3Tc.html", "txt": "/transcripts/7BcQdtRA3Tc.txt", "vtt": "/transcripts/7BcQdtRA3Tc.vtt" } }