{ "video": { "id": "htod7Nv1cBc", "title": "How MiniMax M3 Was Built: Sparse Attention and Native Multimodality — Olive Song", "duration": 903, "upload_date": null, "channel": "AI Engineer", "source": "AI Engineer", "event_format": "" }, "analysis": { "video_id": "htod7Nv1cBc", "title": "How MiniMax M3 Was Built: Sparse Attention and Native Multimodality — Olive Song", "one_liner": "MiniMax M3 is an open-weight model that is natively multimodal from step zero and uses MiniMax Sparse Attention (MSA). MSA keeps a separate token selection for each KV head and retrieves blocks rather than tokens to fit GQA, giving 9x faster prefill and 15x faster decode than full attention at 1M context.", "summary": "Olive Song, who works on reinforcement learning research at MiniMax, describes three things M3 combines that she says no other open-weight model has: SOTA coding and agent capability, a 1M context window via MiniMax Sparse Attention, and native multimodality trained from scratch. She argues MSA comes from co-designing the algorithm and the infrastructure. DeepSeek's DSA is tuned for MQA with a large head dimension. Applied to GQA, it removes the diversity between KV heads, reads GPU memory inefficiently, and makes the top-k step expensive. MSA fixes this with per-KV-head selection and block-level retrieval. On multimodality, she reports that bringing in vision after pretraining (CPT-style) or partway through pretraining either hurt text quality or depended too much on hyperparameters, while training multimodal from scratch barely affected text at the same text token budget.", "key_points": [ "M3 is open weight and claimed to be the first open-weight model to combine SOTA coding/agent ability, a 1M context window and native multimodality. She says it can 'drop in for 90% of the coding works' and do automatic task decomposition.", "MSA has two branches. An index branch runs a lightweight full attention (4 query heads, 1 key head) and is trained with a KL loss to match the main branch's attention scores. It picks the highest-scoring blocks, and the sparse branch attends only to those keys and values. Result: 9x faster prefill and 15x faster decode than full attention.", "Problems with applying DeepSeek's DSA directly to GQA: (1) every KV head attends to the same selected tokens, which removes GQA's diversity; (2) GQA's split layout (e.g. 4x128 instead of MQA's 1x512) means several separate memory reads instead of one contiguous read, lowering the compute-to-memory-access ratio; (3) top-k has inherent data dependencies, so it is hard to parallelise across many token-level candidates.", "MSA's fixes: drop the step that merges index-head outputs so each KV head can attend to different tokens; switch from token-level selection to block-level retrieval, which means fewer candidates, cheaper top-k and merging, and better memory access; and write custom kernels for training and inference. She says this causes no significant performance drop.", "Three multimodal strategies tested at scale: adding vision after the pretraining decay stage (CPT-style) hurt text because the model had largely converged; adding it before the decay stage was very sensitive to data mixture and learning rate, and findings didn't carry over between runs; native multimodal from scratch 'almost does not hurt text capability' at the same text token budget.", "Attention-map analysis: in CPT-style models, text tokens pay very little attention to visual tokens (dark regions on the map). With native training the map is integrated, with no clear separation between text and visual tokens.", "The architecture is standard: a ViT turns images and video into visual tokens, which replace image placeholders in the text sequence before going into the LM. The key change is 3D attention inside the ViT, so patches from different frames attend to each other. This improves video understanding and also transfers to some image benchmarks.", "3D attention can't cover unlimited frames because of system and load-balancing problems. Four frames gave the best trade-off between performance and efficiency." ], "takeaways": [ "Co-design sparse attention with your own attention layout (GQA vs MQA) and the hardware rather than copying an existing design such as DSA.", "Use block-level rather than token-level retrieval in sparse attention to cut top-k cost and improve contiguous GPU memory access.", "With GQA, keep a separate sparse selection for each KV head so the diversity between heads is preserved.", "To add vision without losing text quality, train multimodal from step zero rather than bolting it on through CPT after or during pretraining. Check fusion with attention maps.", "Consider 3D cross-frame attention in the ViT, limited to a small window (MiniMax found 4 frames best), to strengthen visual representations overall." ], "topics": [ "sparse attention", "long context", "multimodality", "model architecture", "pretraining", "open weights", "gpu efficiency", "vision transformers" ], "tools": [ "MiniMax M3", "MiniMax", "MiniMax Sparse Attention (MSA)", "DeepSeek", "DSA (DeepSeek Sparse Attention)", "GQA", "MQA", "ViT" ], "quotes": [ { "text": "Our core design principle is that a simpler architecture can always lead to better scaling properties.", "at": "02:39", "url": "https://www.youtube.com/watch?v=htod7Nv1cBc&t=159s" }, { "text": "This essentially cancels out the diversity that GQA is is supposed to provide.", "at": "05:34", "url": "https://www.youtube.com/watch?v=htod7Nv1cBc&t=334s" }, { "text": "Under the same text token budget, we find that the this approach almost does not hurt text capability.", "at": "11:15", "url": "https://www.youtube.com/watch?v=htod7Nv1cBc&t=675s" }, { "text": "In our exploration, using four frames gave us the best best trade-off between performance and efficiency.", "at": "14:16", "url": "https://www.youtube.com/watch?v=htod7Nv1cBc&t=856s" } ], "words": 2189 }, "summary_url": "/#htod7Nv1cBc", "transcript": { "html": "/transcripts/htod7Nv1cBc.html", "txt": "/transcripts/htod7Nv1cBc.txt", "vtt": "/transcripts/htod7Nv1cBc.vtt" } }