WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=htod7Nv1cBc

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/htod7Nv1cBc.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.640 --> 00:00:13.680
Um all right.

s3
00:00:13.680 --> 00:00:15.360
Um so, hi everyone.

s4
00:00:15.360 --> 00:00:16.960
I My name is Olive.

s5
00:00:16.960 --> 00:00:20.480
I work at Minimax on reinforcement learning research.

s6
00:00:20.480 --> 00:00:36.720
And today, I'm going to talk a little bit about um our new model, Minimax M3, and how we trained it natively multimodal from step zero, and how we trained it with MSA, which is called uh which is Minimax sparse attention.

s7
00:00:36.720 --> 00:00:40.480
Um a little bit introduction of our team.

s8
00:00:40.480 --> 00:00:52.680
Uh we are one of the few labs over the world that uh works on multimodalities, which means that we have published our um of course LLM models before.

s9
00:00:52.680 --> 00:00:55.880
Um we worked on our speech models.

s10
00:00:55.880 --> 00:01:01.920
We have our own video generation models, and we have our coding agent models.

s11
00:01:03.840 --> 00:01:10.960
And lately, we just released our Minimax M3, which is open weight uh with frontier performance.

s12
00:01:10.960 --> 00:01:17.880
So, it is actually the first open weight model to combine the following three um frontier capabilities.

s13
00:01:17.880 --> 00:01:23.280
The first one is what we call soda uh coding and agent capabilities.

s14
00:01:23.280 --> 00:01:28.840
The model is able to do a lot of the uh agentic tasks that you imagine.

s15
00:01:28.840 --> 00:01:32.600
Um it it can do automatic task decomposition.

s16
00:01:32.600 --> 00:01:34.880
It can work in your coding agents.

s17
00:01:34.880 --> 00:01:40.240
It can drop in for 90% of the coding works that you are currently doing.

s18
00:01:40.240 --> 00:01:45.040
And the second pillar is that the model has 1 million context window.

s19
00:01:45.040 --> 00:01:56.720
Um so, which is very long that enable like unlocks pretty much base all basic user long user interactions um of like it handles full code bases.

s20
00:01:56.720 --> 00:02:00.040
And we trained it with MSA Minimax sparse attention.

s21
00:02:00.040 --> 00:02:06.840
So, it is nine times faster pre-fill and 15 times faster decode versus full attention.

s22
00:02:06.840 --> 00:02:13.040
And the third one is that the the model was trained native multi-model from step zero.

s23
00:02:13.040 --> 00:02:19.840
Which means that from step from scratch, we combined visual understanding with um language generation.

s24
00:02:19.840 --> 00:02:25.240
So, the model would understand both capabilities very well from the start.

s25
00:02:25.240 --> 00:02:32.880
And uh so that we can build many more applications on that, say computer use agents, right?

s26
00:02:35.000 --> 00:02:39.320
So, first I'll briefly explain what sparse attention does.

s27
00:02:39.320 --> 00:02:46.480
Our core design principle is that a simpler architecture can always lead to better scaling properties.

s28
00:02:46.480 --> 00:02:50.320
This is the principle we have followed throughout the entire design.

s29
00:02:50.320 --> 00:02:55.959
So, our goal is to always pursue a cleaner and simpler architecture.

s30
00:02:56.160 --> 00:02:59.239
The sparse attention consists of two branches.

s31
00:02:59.239 --> 00:03:02.280
The first one is what we call the index branch.

s32
00:03:02.280 --> 00:03:06.400
Which performs a lightweight full attention computation.

s33
00:03:06.400 --> 00:03:09.400
It uses four query heads and one key head.

s34
00:03:09.400 --> 00:03:15.280
And it is trained with a KL loss to match the attention scores of the main branch.

s35
00:03:15.280 --> 00:03:22.440
The index branch is used to identify the tokens with the highest attention scores.

s36
00:03:22.440 --> 00:03:27.920
These selected tokens are then attended by by the main branch.

s37
00:03:28.000 --> 00:03:32.680
In other words, the sparse branch does not attend to all of the tokens.

s38
00:03:32.680 --> 00:03:34.920
But only to a subset of them.

s39
00:03:34.920 --> 00:03:40.880
This subset is selected through the lightweight computation of the index branch.

s40
00:03:43.120 --> 00:03:54.080
Then, the sparse branch would use the blocks selected by the index branch to to determine which keys and values are needed for attention computation.

s41
00:04:02.720 --> 00:04:04.800
All right.

s42
00:04:04.800 --> 00:04:12.760
And what I want to emphasize today is that our design is not purely algorithm like algorithm driven.

s43
00:04:12.760 --> 00:04:19.640
Instead, we took an algorithm algorithm and infrastructure co-design approach.

s44
00:04:19.640 --> 00:04:22.960
Um a good good example to start with is DSA.

s45
00:04:22.960 --> 00:04:34.440
DSA was a very strong design, but it is also highly optimized for Deep Seek's own architecture, especially the combination of MQA and the large head beam.

s46
00:04:34.440 --> 00:04:43.800
However, when we move to a more commonly used GQA architecture, directly applying the same design becomes less straightforward.

s47
00:04:43.800 --> 00:04:51.040
It introduces several practical issues, especially from the infrastructure and implementation perspective.

s48
00:04:51.080 --> 00:04:53.680
So, this motivates our design choice.

s49
00:04:53.680 --> 00:05:03.840
Instead of simply borrowing an existing sparse attention design, we re-think the architecture under the constraints of GQA and real system efficiency.

s50
00:05:07.360 --> 00:05:13.640
So, what are the issues if we directly apply the existing sparse attention architecture?

s51
00:05:13.640 --> 00:05:24.360
First, from the algorithm perspective, GQA has multiple KV heads, which should ideally provide more diversity across different KV groups.

s52
00:05:24.360 --> 00:05:34.120
However, if we directly reuse the same token selection strategy, multiple KV heads will end up attending to exactly the same selected tokens.

s53
00:05:34.120 --> 00:05:40.080
This essentially cancels out the diversity that GQA is is supposed to provide.

s54
00:05:40.080 --> 00:05:47.880
Second, from the infrastructure perspective, this design is not very friendly to GPU memory access.

s55
00:05:47.880 --> 00:05:51.960
GPUs are much more efficient when reading contiguous data.

s56
00:05:51.960 --> 00:06:01.720
In the MQA setting, we can think of the KV 8 layout as one KV head with a large head dimension, for example, 1 by 512.

s57
00:06:01.720 --> 00:06:06.080
This is relatively contiguous and efficient to read.

s58
00:06:06.080 --> 00:06:11.640
However, in GQA, the same total dimension may be split across multiple KV heads.

s59
00:06:11.640 --> 00:06:15.400
For example, it's 4 by 128.

s60
00:06:15.400 --> 00:06:18.160
These are no longer one single contiguous read.

s61
00:06:18.160 --> 00:06:25.000
Instead, the GPU needs to perform multiple independent reads across different KV heads.

s62
00:06:25.000 --> 00:06:31.960
This increases memory access overhead and reduces the compute-to-memory access ratio.

s63
00:06:32.320 --> 00:06:35.960
And another issue here is the top-k computation itself.

s64
00:06:35.960 --> 00:06:44.280
To select the most important tokens, the indexer needs to compute top-k scores over a large number of token-level candidates.

s65
00:06:44.280 --> 00:06:48.480
However, top-k has inherent data dependencies.

s66
00:06:48.480 --> 00:06:52.400
So, it is difficult to fully parallel on GPU.

s67
00:06:52.400 --> 00:06:59.400
As a result, token-level selection introduces non-trivial computation and merging overhead.

s68
00:06:59.640 --> 00:07:03.800
So, how did we address these issues in MSA?

s69
00:07:03.800 --> 00:07:10.840
From the algo- From the algorithm side, we remove the aggregation of the index branch outputs.

s70
00:07:10.840 --> 00:07:19.600
In the original design, the results from all index branch In index branch heads are merged into a single set of selected tokens.

s71
00:07:19.600 --> 00:07:25.440
But, as we discussed earlier, this would remove the diversity brought by GQA.

s72
00:07:25.440 --> 00:07:30.400
Instead, we keep the selection different different across KV heads.

s73
00:07:30.400 --> 00:07:37.320
Under our sparse attention design, when using GQA, different KV heads can attend to different sets of tokens.

s74
00:07:37.320 --> 00:07:44.720
This helps preserve the diversity of GQA rather than forcing all heads to share the same sparse tension.

s75
00:07:44.720 --> 00:07:50.760
For the infrastructure side, we move from token level selection to block level retrieval.

s76
00:07:50.760 --> 00:08:03.000
Compared with token level selection, block block level retrieval significantly reduces the number of candidates and lowers the overhead of top key computation and result merging.

s77
00:08:03.000 --> 00:08:08.480
At the same time, block level retrieval is more hardware friendly friendly.

s78
00:08:08.480 --> 00:08:14.360
It improves memory access efficiency and increases the compute to memory access ratio.

s79
00:08:14.360 --> 00:08:21.080
Importantly, we also show that the design does not introduce a significant performance drop.

s80
00:08:21.640 --> 00:08:27.080
Finally, we design efficient kernels for both training and inference.

s81
00:08:27.080 --> 00:08:37.280
I I I won't go into the kernel details here today, but this is the system level component that makes the sparse attention design practical and efficient.

s82
00:08:37.280 --> 00:08:46.240
So, with this design, we were able to extend the training of our M3 model to 1 million context very efficiently.

s83
00:08:56.640 --> 00:09:01.640
Um and as we talked about earlier, the third pillar is the native multi-modality.

s84
00:09:01.640 --> 00:09:08.480
Our model, Minimax M3, was trained from scratch with multi-modal data.

s85
00:09:08.600 --> 00:09:09.920
What does that mean?

s86
00:09:09.920 --> 00:09:19.400
Our model was trained with both text and vision understand standing from step zero, which is very different approach from current open weight models.

s87
00:09:19.400 --> 00:09:21.880
Our starting point is very clear.

s88
00:09:21.880 --> 00:09:28.200
At the current stage, a standalone VL model has relatively limited application scenarios.

s89
00:09:28.200 --> 00:09:35.600
But combining it with with, you know, text understanding during the training can be challenging.

s90
00:09:35.600 --> 00:09:44.760
So, the key question and challenge there was how can we introduce multimodal capability without hurting the core text performance?

s91
00:09:45.360 --> 00:09:50.960
To answer this, we actually explored several training strategies at scale.

s92
00:09:51.120 --> 00:10:01.840
The first and most common approach is to add multimodal training after the pre-training decay stage, which is essentially a CPT style approach.

s93
00:10:01.840 --> 00:10:07.640
However, at that point, the model has already largely converged.

s94
00:10:08.000 --> 00:10:16.600
Adding multimodal data at this stage tends to interfere with the learned text capability and causes degradation.

s95
00:10:16.920 --> 00:10:21.840
Another approach is to enter multimodal training before pre-training the decay stage.

s96
00:10:21.840 --> 00:10:28.120
For example, after the model has already seen a number of amount of text tokens.

s97
00:10:28.120 --> 00:10:39.120
We also tried this direction, but found that the results were highly sensitive to the data mixture and training hyper parameters, such as learning rate, for example.

s98
00:10:39.120 --> 00:10:49.120
In one setting, a smaller learning rate might reduce the performance drop, but this conclusion does not necessarily generalize to another run.

s99
00:10:49.120 --> 00:10:57.839
More importantly, during large-scale pre-training, these parameters cannot be arbitrarily changed just for multimodal training.

s100
00:10:58.640 --> 00:11:04.880
And this led us to our final strategy, native multimodal training from scratch.

s101
00:11:04.880 --> 00:11:15.320
In this setting, multimodal capability is introduced from the very beginning rather than being added after the text model has already formed its structure.

s102
00:11:15.320 --> 00:11:23.560
Under the same text token budget, we find that the this approach almost does not hurt text capability.

s103
00:11:26.120 --> 00:11:31.280
We actually did many fun analysis during these experiments.

s104
00:11:31.280 --> 00:11:37.400
Um one of them is this interesting observation uh from the attention maps.

s105
00:11:37.920 --> 00:11:47.840
You can see from this picture that for CPT-style multimodal models, the text tokens often pay very little attention to visual tokens.

s106
00:11:47.840 --> 00:11:52.600
The visual token regions are mostly dark in the attention map.

s107
00:11:52.600 --> 00:11:57.600
This suggests that the multimodalities are not fully integrated.

s108
00:11:58.240 --> 00:12:05.640
Well, in contrast with native multimodal training, the attention map looks much more integrated.

s109
00:12:05.720 --> 00:12:12.080
We do not see here a clear separation between text tokens and visual tokens.

s110
00:12:12.080 --> 00:12:19.360
This suggests that the model learns to fuse visual and textual information more naturally from the beginning.

s111
00:12:19.600 --> 00:12:25.520
So, overall, our training strategy is not just about adding vision capability.

s112
00:12:25.520 --> 00:12:33.560
It is about integrating multimodality into the model while preserving its strongest text capabilities.

s113
00:12:36.600 --> 00:12:44.240
And of course, uh we did many experiments on multimodal architecture choices.

s114
00:12:44.240 --> 00:12:47.520
Our overall architecture is relatively standard.

s115
00:12:47.520 --> 00:12:53.480
The image or video is being processed by a ViT, which produces visual tokens.

s116
00:12:53.480 --> 00:13:02.320
These visual tokens are then inserted into the language model input by re- replacing the original image placeholders in the text sequence.

s117
00:13:02.320 --> 00:13:09.360
After that, the visual embeddings and text embeddings are fed together into the language model.

s118
00:13:09.520 --> 00:13:17.960
One important finding is that applying 3D attention inside the ViT is very helpful for vision understanding.

s119
00:13:17.960 --> 00:13:24.480
In a standard ViT setting, attention is is applied within each individual image or frame.

s120
00:13:24.480 --> 00:13:32.080
For video inputs, this means that visual patches from different frames do not directly interact at the ViT stage.

s121
00:13:32.080 --> 00:13:38.400
But with 3D attention, patches across different frames can attend to each other directly.

s122
00:13:38.400 --> 00:13:45.000
This gives the model stronger temporal and cross-frame visual modeling capability.

s123
00:13:45.280 --> 00:13:52.920
And we found that this improves visual understanding, and the benefit also transfers to some image benchmarks.

s124
00:13:52.920 --> 00:14:02.480
Suggesting that it strengthens the overall visual representation rather than only improving video-specific tasks.

s125
00:14:03.440 --> 00:14:16.720
And one practical detail is that we cannot simply extend 3D attention to an unlimited number of frames because that would introduce additional system and load balancing challenges.

s126
00:14:16.720 --> 00:14:23.760
In our exploration, using four frames gave us the best best trade-off between performance and efficiency.

s127
00:14:24.320 --> 00:14:39.400
So, overall, with our architecture design, with our training strategies that we explored throughout the stages, and with our analysis and ViT explorations, we were able to train the model from scratch

s128
00:14:39.400 --> 00:14:41.960
with multimodality understanding.

s129
00:14:41.960 --> 00:14:44.520
And that would be the end of the presentation today.

s130
00:14:44.520 --> 00:14:46.800
Thank you.

s131
00:14:59.325 --> 00:15:01.325
[music]