Why AI Agents Need Million-Token Context — Thomas Wolf & Olive Song, MiniMax

AI Engineer · 20 min · 257 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

1. 00:01[music]
2. 00:12Joining us on stage is the co-founder and chief science officer at Hugging Face, Thomas Wolf.
3. 00:20[music]
4. 00:26[music]
5. 00:32Hello everyone.
6. 00:33Hello Olive, nice to have you on stage.
7. 00:36Hi, nice to meet you.
8. 00:37Thanks for having me, yeah.
9. 00:38So I think you're on for a treat today because you just saw a GLM uh which is current number two on the artificial intelligence leaderboard.
10. 00:48I take Fable out because nobody can use it.
11. 00:51And now we have number four.
12. 00:52So you basically you will have all the top models, at least the top open source model in a row.
13. 00:57And we're very lucky to have Olive who has a pretty amazing path in life.
14. 01:04Uh so she came to the US, Pennsylvania.
15. 01:06She was studying, doing PhD at uh NYU uh in the lab of Jan LeCun working on J Pa, but we decided we won't talk about J Pa today, right?
16. 01:17Something for another day.
17. 01:19Um and then instead of joining Hugging Face, which was in New York also at that time she decided to go join MiniMax.
18. 01:28So for those who who maybe don't know all the all the neo labs around the world and you're you're forgiven because I think there's like 64 neo labs right now.
19. 01:38MiniMax is one of the top of what we call the AI dragons in China.
20. 01:43So these are the new there's there's Deep Seek which is very well known now, Moonshot who does Kimi Z and GLM that you just saw and now we have a MiniMax.
21. 01:53They're all extremely good, extremely a team uh fighting for the first spot.
22. 01:59Uh so the the the latest uh release of MiniMax was M3 uh just earlier earlier in June, which was the the top model at the time, top open-source model.
23. 02:09Uh very impressive.
24. 02:10There's a lot of very interesting things about this model, so we'll quickly dive in them.
25. 02:15And then talk a little bit about uh what's what's what's specific about MiniMax, what's what's great there.
26. 02:22So uh maybe Olive to to start a little bit.
27. 02:25Can you Can you give us, you know, a a little bit of your your view of of M3, what you like about this model, how was the release?
28. 02:33Mhm.
29. 02:34Yeah, M3 we released M3 earlier this month, and it is a smaller model with 400 around 400 billion total parameters and 20 billion activated.
30. 02:45Um but it is very capable in terms of both coding performances, and also it understands vision.
31. 02:52So um that's uh what open-source models don't usually have.
32. 02:58It's that they can the model can only deal with coding, but it can also understand videos, um images, and it has a super uh long context of 1 million.
33. 03:09Um with our new architecture called MSA, MiniMax Sparse Attention.
34. 03:14So we we really put these three things together uh because we know that they are they will be very important in future AI applications.
35. 03:25Coding capabilities, agentic capabilities, longer context, and multimodal understanding.
36. 03:32Um yeah, I think that would be very interesting about the model.
37. 03:35Yeah, so so there's a lot to unpack unpack in this model, and it's um it's it's still, I think, the only top five model open-source model that is actually multimodal, so we need to talk about that.
38. 03:46But maybe first about the long context, because that was also the first one that really had this real 1 million token long context is actually functional and you guys had also the the Minimax pass attention
39. 03:59which is this one technique to to make that efficient that you also published and and share extensively.
40. 04:05So can you can you talk a little bit about this?
41. 04:07Maybe how the project went from from the attention how to make this long context.
42. 04:12Yeah, I would say the story about long context went back to even Minimax M1 and Minimax 01 where the model was actually was able to perform tasks of 10 million
43. 04:25token context.
44. 04:2610 million?
45. 04:2610 million, yes.
46. 04:28But then it was not an agentic model, right?
47. 04:31It was just 10 for example, you dump in a book, it would be able to give reviews on it and stuff like that.
48. 04:37So what we realized was that, you know, longer context actually unlocks a lot of capabilities especially when interacting with users and now when, you know, the agent is interacting with the whole environment and getting all the tool responses,
49. 04:54getting multi rounds the like shorter context wouldn't be enough to perform the complex task.
50. 05:04So for this version we said, "Oh, we have to have our longer context backs."
51. 05:10So what we pursued was with our Minimax sparse attention.
52. 05:15Um which you know, was the architecture that was scalable and had a simple design.
53. 05:24So I would say from a higher level, right?
54. 05:27It has an index branch that, you know, selects on a higher level what is what matters more in the context and then we have a sparse attention branch that calculates performs the calculation on the selected blocks
55. 05:43to actually performs the task.
56. 05:45Um and so yeah, like that we really designed um an elegant architecture so that we can scale the length and then scale the model size in the future with that.
57. 05:57That's beautiful.
58. 05:58I like how for those who've been in the field for quite some time, we we had a lot of work on attention, right?
59. 06:03This N square and there was a lot of linear attention.
60. 06:06Yeah.
61. 06:06And then some that somehow all of these disappeared at some point when flash attention came around.
62. 06:11We discovered we just needed more efficient camera.
63. 06:13Now, I like how we come back to thinking, you know, first principle, what is attention?
64. 06:18How can we make that more efficient?
65. 06:20So, 1 million token is crazy, right?
66. 06:23GPT-2 was 1,024 and and everyone was like, "Oh, that's really big.
67. 06:27We we we never need more."
68. 06:29Where do you see this coming?
69. 06:30Like, going in the future?
70. 06:31Like, Jeff Dean was pitching me the other day a trillion token attention.
71. 06:35You think we should go to a trillion token attention?
72. 06:38So, that's definitely something we can explore towards, right?
73. 06:42Ultra lengths of the context.
74. 06:44Definitely, it's something that's very exciting to explore with.
75. 06:47And something that architecture design along with hardware um would require a lot of research onto that.
76. 06:54Yeah.
77. 06:54You think there's still a lot of low-hanging fruits?
78. 06:57So, typically today we saw Open AI really reducing I mean, we don't know how as of firm, but like reducing their their inference bill by half, by probably having some more efficient processing around tensions of any type.
79. 07:09You think there's still a lot of low-hanging fruit that can be getting um how we can process that.
80. 07:14So, so one one thing we're still very interesting about M3 is how cheap it is in particular because of this part attention and part because of its small one, but it's also very efficient.
81. 07:23Right.
82. 07:23You think we can go even way further?
83. 07:25And maybe how did you guys invented uh Min Max Fast Attention?
84. 07:30Was it an agent coming up with the idea?
85. 07:32Was it a human still coming up with the idea?
86. 07:34Tell us a little bit about
87. 07:35Yeah.
88. 07:36Um, so we do think there's still a lot of work that can get into architecture and inference optimization so that the model can be more efficient, especially if there are tasks that are very task sensitive
89. 07:49but require very strong capabilities, right?
90. 07:52And for that those kind of tasks we really want model to be efficient.
91. 07:57Um and who came up with this part of actually I think an intern from our team worked on that.
92. 08:03That yeah, an intern.
93. 08:05Uh that doesn't usually happen in a lot of labs because I think in some labs interns don't have access to the data, the work, and stuff.
94. 08:15Uh but yeah, we are open to anyone who would like to contribute to our models.
95. 08:20So, um the architecture was actually designed by an intern.
96. 08:24It's very good.
97. 08:25Still some work for interns here.
98. 08:27Good good news.
99. 08:28Um that's also a good segue to also how Min Max is working internally.
100. 08:32So, so we were discussing before coming on stage I was saying everyone can propose a project.
101. 08:38Can you tell us a little bit about how you are organized, how you do your research?
102. 08:42Mhm.
103. 08:42Mhm.
104. 08:43I think that is very different from uh even in school or even in earlier you know, the earlier tech companies is pretty pretty different.
105. 08:54It's that um what we what we make sure is that we have good foundation and good um infrastructure so that anyone can play with the model and can think of what they can improve with the model.
106. 09:08And then after model releases when they are free, right?
107. 09:11They can play with the model.
108. 09:13They can think of their own evaluations.
109. 09:16They can find their own weaknesses and propose a thing that they want to improve on the model.
110. 09:21And then other people who are interested in that would, you know, propose to join the project and they will work on for a couple of weeks or even a couple of months.
111. 09:30And when they work out, the final thing is shipped to our model.
112. 09:34It it is, you you we use that in our final training and it's shipped out to the audience.
113. 09:39Interesting.
114. 09:39So, you can have people working for a really long time on project.
115. 09:42When you say a couple of months, it can be like really deep exploration of
116. 09:46Yes.
117. 09:46Yes.
118. 09:48I would say, for example, architecture might require longer time of investigation, research, experiments, even redoing the evaluations for pre-training.
119. 09:58Yes, so it might require longer time.
120. 10:00Very nice, yeah.
121. 10:01And I know you're also very big on evaluation.
122. 10:03I agree.
123. 10:03We could talk about that.
124. 10:04I think what One thing probably related to that is this unique specificity that M3 and your team has um around multimodality.
125. 10:12So, not just text, but the similar can also understand image and video.
126. 10:16And as I understand, but but please explain explain better, when you read the model card on Hugging Face, it say the model was trained from the first step as a multimodal, not just have a like user one as after solved, right?
127. 10:28Can you tell us a little bit more about that and why you think it's important and and and why starting from the first step on multimodal training and not just just training this.
128. 10:37Um so, we call it native multimodality.
129. 10:40Um and so, it is somehow typical for model labs to train the multimodal, let's say, vision understanding capabilities after the text pre-training is done.
130. 10:54Um they put adapters and then train that part.
131. 10:57But what we found out was that that would actually harm the text performance.
132. 11:03And the vision vision understanding performance wouldn't converge that well because the model is kind of converges towards the model the text understanding.
133. 11:12Um and it's just not the most optimal.
134. 11:15And also not the most scalable, if you think about it.
135. 11:18We want to scale the data, right?
136. 11:20And also, we can also some labs um train this capability from halfway through the pre-training.
137. 11:27For example, continued pre-training.
138. 11:30But what we found that this would be very, you know, uh recipe sensitive.
139. 11:36It is different for the recipe would be different for different architectures, different, you know, data mixtures, different learning rates.
140. 11:44It's hard to control, hard to, you know, scale to you can't really scale your experiment results and conclusions to a larger model.
141. 11:55And so you know, what we thought was why not just training from the very first step?
142. 12:01That comes to the most natural.
143. 12:04We know that a lot of labs run into problems doing that.
144. 12:08The model would collapse after a couple of steps of training, you know, both text and vision understanding, but we managed to solve that problem.
145. 12:18We did a lot of work on VIT and we did a lot of work on the data that we actually training.
146. 12:26For example, we do interleave the data, what we call interleave the data.
147. 12:31It's actually natural data, but we keep the images and videos in instead of masking it out and we do some pretty good cleaning and masking on the data and we do very good reward modeling
148. 12:45so that we train it from the first step and scales up a lot.
149. 12:50Yeah, it does does not collapse.
150. 12:53That's really impressive.
151. 12:54Impressive.
152. 12:55Should we Should we expect much larger model in the future?
153. 12:58So this one is still fairly small, right?
154. 13:00It's It's 428 billion parameters, 23 active billion.
155. 13:05Well, do you think you will go past the trillion?
156. 13:09Definitely.
157. 13:10Yeah, definitely in the future.
158. 13:13There are many tasks that wouldn't be able to the more model wouldn't be able to perform very good at with smaller parameters.
159. 13:22We are definitely going more ambitious than this.
160. 13:25It's great.
161. 13:26Looking forward.
162. 13:27Um another interesting thing I I always been find fascinating about Min Max is how how you also have this whole range of of apps and product, right?
163. 13:36So, I remember already So, so Min Max started to open source things on the on the hugging face platform in in January last year.
164. 13:43So, that 18 month ago and and we were chatting a little bit about the team to understand what you were doing and I remember you So, you were already having a huge usage
165. 13:52on some of these of some of these apps.
166. 13:55Um can you tell us a little bit how how this started, right?
167. 13:58So, was it basically you had a lot of apps and then you thought we have we have all this data, why not training a model and then they build up research team.
168. 14:05How is how is the story there?
169. 14:07Um our our story is modeled from the first day.
170. 14:12So, um I believe that multi-modality model a model that can understand all visions and outputs all modalities was the first thing that our um CEO planned on the first day even before the company even started.
171. 14:26So, that was the dream of AGI.
172. 14:27I think that was very very early even before ChatGPT came out.
173. 14:31Wow.
174. 14:31Um yeah.
175. 14:33And then apps were something that comes along because you have some model capabilities you want people to experience it well.
176. 14:40Not many people can use it with API, right?
177. 14:43We can't expect everyone to experience with API.
178. 14:46So, we need good um user interaction, you know, interfaces, good apps, good scenarios that people can can you know, experience experience model with.
179. 14:58I think actually those apps covered more than 300 million people around 200 countries globally.
180. 15:07And I think over a million companies as well.
181. 15:10Yeah, this was a mind-blowing when I heard about the the size and we we don't often realize the size of of of this type of usage already.
182. 15:17And and that I kind of brings me to the question around um open source business model, and and all of that, which is the always existing question, which is right now it's nice to open source model, but you you also need to have some revenue stream, right?
183. 15:34So, I guess M3 is something you you decided, for instance, to be for free, and I'm I think it's it's it's great for the world.
184. 15:41Um how do you see this?
185. 15:42Do you also have some specific models you use for the app?
186. 15:45Do you think about Do you think in the future you'll keep It's probably hard to say for sure, but do you think you'll keep open sourcing models?
187. 15:52How is the culture around open sourcing right now?
188. 15:55Personally, and also for the model research team, we always hope to open source the models.
189. 16:01Um that is our plan.
190. 16:03Because we really see how the open source community together can help the model build better.
191. 16:09For example, we receive a lot of um feedbacks on the model performance from the great community, and we receive PRs on uh wh- whatever we open source.
192. 16:20And those are very very valuable and come comes to our later versions.
193. 16:24So, definitely open sourcing is great.
194. 16:27That's great.
195. 16:28And actually, do you have some ask for the audience, people who are using uh M3 or MiniMax?
196. 16:33Is there something you would love them to send back to you as feedback?
197. 16:38Do you Do you, for instance, do you read when people try to modify the models or play around, you know, tweaks?
198. 16:44Or what is the best thing you you think you can take from the community uh for future models, for instance?
199. 16:50Mhm.
200. 16:51I would say whatever um issues that people are running into, especially with multimodality, right?
201. 16:57This is the first time that we're combining it together.
202. 17:00We are definitely going more ambitious on that in the future.
203. 17:03It might have some flaws right now, but we are improving on that.
204. 17:06So, whatever that's uh feedback that model is not good doing that great, we will definitely improve that in future versions.
205. 17:13And also, whatever features that uh people want.
206. 17:17Say, you know, for example, thinking effort.
207. 17:21Right?
208. 17:21Some people ask for that.
209. 17:23Um like everyone can ask, and we will try to accomplish that in the future models.
210. 17:29Yeah.
211. 17:29Do you see a lot of usage right now already in multimodality in terms of coding agents?
212. 17:34I feel like it's it's a little bit un- underexplored.
213. 17:38It is It is, but um it can actually unlocks a lot of capabilities and a lot of uh agent applications.
214. 17:47Say that for example, you want a model to read a PP- PowerPoint, uh or to read some report that is not very structured.
215. 17:55Um and you want it to understand a very long video, say that you dump in a long-playing video, and then you want the model to act uh using some tools
216. 18:05uh after understanding it.
217. 18:07And it unlocks a wide variety of um agent use cases.
218. 18:12So, like the agent could finally watch my YouTube tutorial and understand how to use my coding tools, how I described it?
219. 18:18Is it something like that?
220. 18:20Uh
221. 18:21Could the agent finally watch YouTube tutorials and understand things from them?
222. 18:25Yeah, yeah, yeah.
223. 18:26I think so.
224. 18:27Do you use a lot of uh agent coding tools internally?
225. 18:30Is it like I mean, coding for sure, but like is it also already in terms of research?
226. 18:35Is it automated part or not?
227. 18:37How How does this uh
228. 18:38Yes.
229. 18:39work?
230. 18:39Um we have our own research harnesses.
231. 18:42Um we build our own research harnesses that automate our workflows.
232. 18:46I would say a lot of our workflows are automated.
233. 18:49You can see how um the latest frontier models all pursues capability less kernel optimization, right?
234. 18:57Like let the model post string other models.
235. 19:01Um let the model build data.
236. 19:03Auto data, stuff like that.
237. 19:05Um you can see how more and more models are capable of doing those, including M3.
238. 19:11Actually, we were very good at those cases, longer horizons and coronal organizations.
239. 19:16Um and so we can use that model capability, harness it together, and help with our um daily routine, and make our iterations even faster.
240. 19:27Is M3 building M4 already?
241. 19:29Um building M3.1.
242. 19:32M3.1, okay.
243. 19:33[laughter]
244. 19:34Let's hit the gym.
245. 19:34Already.
246. 19:36Um I I would love to finish on what what you find exciting in the coming month, what what do you think it it can be Asia in terms of feature
247. 19:45or or things you want to see happening in in AI or more generally in terms of whatever whatever really is top of your mind I would say and it's going to
248. 19:54happen.
249. 19:55A lot of things are very exciting.
250. 19:58Um but what I recently find the most exciting would be a multi-agents that I think a lot of AI applications are using, model routing, multi-agents um that allows even more capabilities,
251. 20:12even more complex tasks, and also it tells us what the models are capable and not capable of, and you can, you know, do a lot of things with that.
252. 20:23It's pretty exciting.
253. 20:24Thanks a lot, Alif.
254. 20:26Pleasure to have you.
255. 20:27Thanks for having me.
256. 20:28Thanks, everyone.
257. 20:30[applause]