How MiniMax M3 Was Built: Sparse Attention and Native Multimodality — Olive Song

AI Engineer · 15 min · 131 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

1. 00:01[music]
2. 00:12Um all right.
3. 00:13Um so, hi everyone.
4. 00:15I My name is Olive.
5. 00:16I work at Minimax on reinforcement learning research.
6. 00:20And today, I'm going to talk a little bit about um our new model, Minimax M3, and how we trained it natively multimodal from step zero, and how we trained it with MSA, which is called uh which is Minimax sparse attention.
7. 00:36Um a little bit introduction of our team.
8. 00:40Uh we are one of the few labs over the world that uh works on multimodalities, which means that we have published our um of course LLM models before.
9. 00:52Um we worked on our speech models.
10. 00:55We have our own video generation models, and we have our coding agent models.
11. 01:03And lately, we just released our Minimax M3, which is open weight uh with frontier performance.
12. 01:10So, it is actually the first open weight model to combine the following three um frontier capabilities.
13. 01:17The first one is what we call soda uh coding and agent capabilities.
14. 01:23The model is able to do a lot of the uh agentic tasks that you imagine.
15. 01:28Um it it can do automatic task decomposition.
16. 01:32It can work in your coding agents.
17. 01:34It can drop in for 90% of the coding works that you are currently doing.
18. 01:40And the second pillar is that the model has 1 million context window.
19. 01:45Um so, which is very long that enable like unlocks pretty much base all basic user long user interactions um of like it handles full code bases.
20. 01:56And we trained it with MSA Minimax sparse attention.
21. 02:00So, it is nine times faster pre-fill and 15 times faster decode versus full attention.
22. 02:06And the third one is that the the model was trained native multi-model from step zero.
23. 02:13Which means that from step from scratch, we combined visual understanding with um language generation.
24. 02:19So, the model would understand both capabilities very well from the start.
25. 02:25And uh so that we can build many more applications on that, say computer use agents, right?
26. 02:35So, first I'll briefly explain what sparse attention does.
27. 02:39Our core design principle is that a simpler architecture can always lead to better scaling properties.
28. 02:46This is the principle we have followed throughout the entire design.
29. 02:50So, our goal is to always pursue a cleaner and simpler architecture.
30. 02:56The sparse attention consists of two branches.
31. 02:59The first one is what we call the index branch.
32. 03:02Which performs a lightweight full attention computation.
33. 03:06It uses four query heads and one key head.
34. 03:09And it is trained with a KL loss to match the attention scores of the main branch.
35. 03:15The index branch is used to identify the tokens with the highest attention scores.
36. 03:22These selected tokens are then attended by by the main branch.
37. 03:28In other words, the sparse branch does not attend to all of the tokens.
38. 03:32But only to a subset of them.
39. 03:34This subset is selected through the lightweight computation of the index branch.
40. 03:43Then, the sparse branch would use the blocks selected by the index branch to to determine which keys and values are needed for attention computation.
41. 04:02All right.
42. 04:04And what I want to emphasize today is that our design is not purely algorithm like algorithm driven.
43. 04:12Instead, we took an algorithm algorithm and infrastructure co-design approach.
44. 04:19Um a good good example to start with is DSA.
45. 04:22DSA was a very strong design, but it is also highly optimized for Deep Seek's own architecture, especially the combination of MQA and the large head beam.
46. 04:34However, when we move to a more commonly used GQA architecture, directly applying the same design becomes less straightforward.
47. 04:43It introduces several practical issues, especially from the infrastructure and implementation perspective.
48. 04:51So, this motivates our design choice.
49. 04:53Instead of simply borrowing an existing sparse attention design, we re-think the architecture under the constraints of GQA and real system efficiency.
50. 05:07So, what are the issues if we directly apply the existing sparse attention architecture?
51. 05:13First, from the algorithm perspective, GQA has multiple KV heads, which should ideally provide more diversity across different KV groups.
52. 05:24However, if we directly reuse the same token selection strategy, multiple KV heads will end up attending to exactly the same selected tokens.
53. 05:34This essentially cancels out the diversity that GQA is is supposed to provide.
54. 05:40Second, from the infrastructure perspective, this design is not very friendly to GPU memory access.
55. 05:47GPUs are much more efficient when reading contiguous data.
56. 05:51In the MQA setting, we can think of the KV 8 layout as one KV head with a large head dimension, for example, 1 by 512.
57. 06:01This is relatively contiguous and efficient to read.
58. 06:06However, in GQA, the same total dimension may be split across multiple KV heads.
59. 06:11For example, it's 4 by 128.
60. 06:15These are no longer one single contiguous read.
61. 06:18Instead, the GPU needs to perform multiple independent reads across different KV heads.
62. 06:25This increases memory access overhead and reduces the compute-to-memory access ratio.
63. 06:32And another issue here is the top-k computation itself.
64. 06:35To select the most important tokens, the indexer needs to compute top-k scores over a large number of token-level candidates.
65. 06:44However, top-k has inherent data dependencies.
66. 06:48So, it is difficult to fully parallel on GPU.
67. 06:52As a result, token-level selection introduces non-trivial computation and merging overhead.
68. 06:59So, how did we address these issues in MSA?
69. 07:03From the algo- From the algorithm side, we remove the aggregation of the index branch outputs.
70. 07:10In the original design, the results from all index branch In index branch heads are merged into a single set of selected tokens.
71. 07:19But, as we discussed earlier, this would remove the diversity brought by GQA.
72. 07:25Instead, we keep the selection different different across KV heads.
73. 07:30Under our sparse attention design, when using GQA, different KV heads can attend to different sets of tokens.
74. 07:37This helps preserve the diversity of GQA rather than forcing all heads to share the same sparse tension.
75. 07:44For the infrastructure side, we move from token level selection to block level retrieval.
76. 07:50Compared with token level selection, block block level retrieval significantly reduces the number of candidates and lowers the overhead of top key computation and result merging.
77. 08:03At the same time, block level retrieval is more hardware friendly friendly.
78. 08:08It improves memory access efficiency and increases the compute to memory access ratio.
79. 08:14Importantly, we also show that the design does not introduce a significant performance drop.
80. 08:21Finally, we design efficient kernels for both training and inference.
81. 08:27I I I won't go into the kernel details here today, but this is the system level component that makes the sparse attention design practical and efficient.
82. 08:37So, with this design, we were able to extend the training of our M3 model to 1 million context very efficiently.
83. 08:56Um and as we talked about earlier, the third pillar is the native multi-modality.
84. 09:01Our model, Minimax M3, was trained from scratch with multi-modal data.
85. 09:08What does that mean?
86. 09:09Our model was trained with both text and vision understand standing from step zero, which is very different approach from current open weight models.
87. 09:19Our starting point is very clear.
88. 09:21At the current stage, a standalone VL model has relatively limited application scenarios.
89. 09:28But combining it with with, you know, text understanding during the training can be challenging.
90. 09:35So, the key question and challenge there was how can we introduce multimodal capability without hurting the core text performance?
91. 09:45To answer this, we actually explored several training strategies at scale.
92. 09:51The first and most common approach is to add multimodal training after the pre-training decay stage, which is essentially a CPT style approach.
93. 10:01However, at that point, the model has already largely converged.
94. 10:08Adding multimodal data at this stage tends to interfere with the learned text capability and causes degradation.
95. 10:16Another approach is to enter multimodal training before pre-training the decay stage.
96. 10:21For example, after the model has already seen a number of amount of text tokens.
97. 10:28We also tried this direction, but found that the results were highly sensitive to the data mixture and training hyper parameters, such as learning rate, for example.
98. 10:39In one setting, a smaller learning rate might reduce the performance drop, but this conclusion does not necessarily generalize to another run.
99. 10:49More importantly, during large-scale pre-training, these parameters cannot be arbitrarily changed just for multimodal training.
100. 10:58And this led us to our final strategy, native multimodal training from scratch.
101. 11:04In this setting, multimodal capability is introduced from the very beginning rather than being added after the text model has already formed its structure.
102. 11:15Under the same text token budget, we find that the this approach almost does not hurt text capability.
103. 11:26We actually did many fun analysis during these experiments.
104. 11:31Um one of them is this interesting observation uh from the attention maps.
105. 11:37You can see from this picture that for CPT-style multimodal models, the text tokens often pay very little attention to visual tokens.
106. 11:47The visual token regions are mostly dark in the attention map.
107. 11:52This suggests that the multimodalities are not fully integrated.
108. 11:58Well, in contrast with native multimodal training, the attention map looks much more integrated.
109. 12:05We do not see here a clear separation between text tokens and visual tokens.
110. 12:12This suggests that the model learns to fuse visual and textual information more naturally from the beginning.
111. 12:19So, overall, our training strategy is not just about adding vision capability.
112. 12:25It is about integrating multimodality into the model while preserving its strongest text capabilities.
113. 12:36And of course, uh we did many experiments on multimodal architecture choices.
114. 12:44Our overall architecture is relatively standard.
115. 12:47The image or video is being processed by a ViT, which produces visual tokens.
116. 12:53These visual tokens are then inserted into the language model input by re- replacing the original image placeholders in the text sequence.
117. 13:02After that, the visual embeddings and text embeddings are fed together into the language model.
118. 13:09One important finding is that applying 3D attention inside the ViT is very helpful for vision understanding.
119. 13:17In a standard ViT setting, attention is is applied within each individual image or frame.
120. 13:24For video inputs, this means that visual patches from different frames do not directly interact at the ViT stage.
121. 13:32But with 3D attention, patches across different frames can attend to each other directly.
122. 13:38This gives the model stronger temporal and cross-frame visual modeling capability.
123. 13:45And we found that this improves visual understanding, and the benefit also transfers to some image benchmarks.
124. 13:52Suggesting that it strengthens the overall visual representation rather than only improving video-specific tasks.
125. 14:03And one practical detail is that we cannot simply extend 3D attention to an unlimited number of frames because that would introduce additional system and load balancing challenges.
126. 14:16In our exploration, using four frames gave us the best best trade-off between performance and efficiency.
127. 14:24So, overall, with our architecture design, with our training strategies that we explored throughout the stages, and with our analysis and ViT explorations, we were able to train the model from scratch
128. 14:39with multimodality understanding.
129. 14:41And that would be the end of the presentation today.
130. 14:44Thank you.
131. 14:59[music]