Is a Faster Model Actually Faster? Test the Whole Job — Morgan Suo, MiniMax

AI Engineer · 17 min · 167 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

1. 00:12Uh I'm Morgan from Minimax.
2. 00:14Uh how many of you knows Minax before?
3. 00:17Oh wow.
4. 00:18I like that.
5. 00:20Yeah.
6. 00:20Because uh I joined this M uh company for three almost three years.
7. 00:24But when we first like step into the American market like uh almost no one knows us.
8. 00:30But at that time we have the first video generation model.
9. 00:33So I will show you something of that at the end of this speech.
10. 00:36But first I will show you like about our the latest language model which will be released next week.
11. 00:43So uh today um I would like to my topic is the hidden cost of a faster model.
12. 00:49So when people thinking about like if you have a AI application your product built in in your own pro workflow.
13. 00:58So when a new model arrives when I say oh please we have a new model do you want to try it?
14. 01:06It can generate a task faster but your I think your question is should I switch to that model?
15. 01:13So you try a research task and the report just comes back very soon and but it misses a very important source.
16. 01:22So someone has to find that source and fix that report.
17. 01:25I think this is the thing we did every every day.
18. 01:29So did the job really get faster with this faster model.
19. 01:35Today we will look at how we reduce the model computation and then we will ask how to test the value of those changes of this model.
20. 01:44So let me show you the job that we will use to answer this question.
21. 01:50So at minimax we build different multimodel models and pro products that people use and a quick chat reply and research report actually have different needs.
22. 02:01So the task tells us what good performance means.
23. 02:05Today I will focus on our latest uh large language model M3.1 flash and one research job.
24. 02:12So here's the analyst question as AI server demand is growing are the companies supplying the circuit boards turning this growth into cash?
25. 02:25The PCB there means printed circuit board.
26. 02:28I think you guys know that.
27. 02:29and unc it connects electronic components that inside devices that such as the AI servers.
28. 02:36So our research case looks like companies in this supply chain.
29. 02:40The internal review describes work across 14 company reports and the model has several jobs.
30. 02:47It must find the right reports.
31. 02:49It must compile the same reporting periods and it must keep different businesses separate.
32. 02:55Then it must explain what the numbers mean.
33. 02:58Think about the analyst who receives the report.
34. 03:02They need to open the source behind an important claim and they need to check a calculation.
35. 03:08They need to see what evidence could change the conclusion.
36. 03:11So a useful report gives the analyst something that they can check and that is our standard for a finished research job.
37. 03:20So when funding shows why those checks matter.
38. 03:26This internal review highlighted a very useful distinction.
39. 03:30So profit can grow but cash flow tells a different story.
40. 03:36A company sells more so is reported pro profit rises but customers have not paid yet or it has uh bought materials that are still in storage.
41. 03:48So the anal the analyst needs to know why the model should point to the cash flow statement and the relevant is disclosures.
42. 03:57So a report earns trust when it claims can be checked and the internal review here describes the research output.
43. 04:05So we still need to verify it sources and calculations and also we also need to measure the time and cost on this same job.
44. 04:12But the technical task a heart use separate tasks.
45. 04:17So they help us to understand specific trade-offs.
46. 04:21With this job clear, so let's look at what changes inside M3.1 flash.
47. 04:28This table shows the main changes inside M3.1 flash.
48. 04:33So I will connect them to a few simple questions.
49. 04:36First, where can we reduce the attention work?
50. 04:40And second, where can we store and move data?
51. 04:45And third, how does the serving setup affect the time a user waits?
52. 04:52And fourth, we also control how much reasoning the model does.
53. 04:57These choices affect different parts of the job and each knows a check against the result we want.
54. 05:04So every optimization has to help the finished job.
55. 05:09And let's begin with how the model uses a long input.
56. 05:15So nowadays I think every model says oh I have a very big context window like we said we have 1 million.
57. 05:22So among this one million how it can help you to find the output find your result quickly faster and correct.
58. 05:31So some passenger passages like in this report like miter just much more than others for this question.
59. 05:39So sparse attention itself means like we uses a very small selection step first like the blue uh box there.
60. 05:48So it scores the blocks of the tax and the main attention then calculation calculate the uses of the selected blocks and the model also keeps the nearby tax in view
61. 06:01and this selection actually they can change with the query and different questions can also needs different passengers of the the context and the number on at the bottom show how much text is published
62. 06:14examples lacks and the full input can be much longer.
63. 06:18For our research job, the goal is to reduce this attention work across many of uh these documents.
64. 06:26But just within this document, a very short footnote there may change the entire conclusion.
65. 06:33So for a cash flow question, the footnote may matter more than the pages of industry outlook.
66. 06:40So we want the model to include that passage too.
67. 06:44less attention work still has to find the important evidence.
68. 06:48So the next question is simple.
69. 06:50Can we still find the facts we need M3.1 flash also uses like the this bar's attention in its first three layers that makes this retrieval an important check.
70. 07:05So this separate task places at different OP positions in long inputs.
71. 07:10So in v run the model passes all 39 requests and the longest inputs were about uh 830k uh tokens.
72. 07:22So the result covers the cases in this test and for our research job we need with real reports.
73. 07:29Can the model find a cash flow footnote?
74. 07:32Does it use the right reporting period?
75. 07:35And can it spot a conflict between different documents?
76. 07:39So move this important passage to different places in the input and then ask the same question again.
77. 07:46A long input is useful when the model can find a key fact.
78. 07:50So finding the evidence is one track too.
79. 07:53Next we look at the numbers we used inside this model.
80. 07:58So the model keeps the internal information from earlier text which we call this KV cache.
81. 08:05And M3.1 flash changes its keys and values from eight bits to four bits, but the query still stays at eight bits.
82. 08:14And this can reduce the data we store and move.
83. 08:17It also changes the position of that internal information.
84. 08:21Look at that table.
85. 08:23So the average scores actually is almost the same but the sign score there it drops like 6.6 percentage.
86. 08:31So some but some other scores rises.
87. 08:35So the average score actually can hide a mistake we your user cares about.
88. 08:42This compares that two full setups but it does not tell us which changes caused each score.
89. 08:48So for our research job we should check the output out output again.
90. 08:53Are the sources right?
91. 08:54Are the calculations right?
92. 08:56And does the explanation match the evidence?
93. 08:59So the same ideas applies to another part of the model is experts.
94. 09:05Uh our model is the MO ine architecture.
95. 09:09So it has the computing parts that we called experts which is the uh M of E experts.
96. 09:17So it selects some of them for each token in M3.1 flash.
97. 09:22These selected experts use four bit ways and act activations.
98. 09:28The shared expert is outside this change and ways of stored model uh values and activations are values the model calculates and lower precision can reduce data movement and support more efficient computation.
99. 09:43But the full calculation has several steps.
100. 09:46So we scale the numbers, we run the calculation and restore the scale.
101. 09:51The position of these steps really matters and the table shows like where scale restore happens and getting that order right is the part of getting the answer right.
102. 10:02So fast math still has to be corrected.
103. 10:06And when we check the complete result, we we measure the complete imp uh implementation.
104. 10:11And for this research job, we run the same questions again.
105. 10:15We inspect why the answers still miss our checks.
106. 10:19And now we look at insight.
107. 10:22We have already look at inside the model.
108. 10:24And now we can look at how it generates an answer for you.
109. 10:30So a serving method called speculative decoding works in two stages.
110. 10:36A faster draft process.
111. 10:39It can help you to propose a short sequence of tokens and the then the targeting model can tracks it and the system keeps this first part that the target accepts and makes a correction when it's needed
112. 10:54and then continues this whole process.
113. 10:57And sometimes it keeps several proposed tokens and sometimes the draft does work what it cannot keep.
114. 11:04So the opportunity here is to get several tokens from one target check and but drafting takes time checking takes time.
115. 11:12Rejecting this tokens also takes time.
116. 11:14So count the time spent checking the draft too.
117. 11:18These checks propose tokens and but financial claims and sources need their own checks.
118. 11:25So for our research job we should also include the source retrieval the tool calls and retries.
119. 11:32The final measure is the time to a report that an analyst can really use in their home own work.
120. 11:39So even when one request runs well many users can change this diff experience.
121. 11:47Here's an separate serving test.
122. 11:50So it's uses the storage generation with low reasoning effort but the concurrence requests now like we rise from the 32s to 64.
123. 12:01So the total output per second grows about 1.9 times but look at the weight time actually it the P95 weight for the first token rises.
124. 12:15It goes from about 1.5 seconds to 11.3 seconds uh6 seconds.
125. 12:22So for 95% of this request the first token arrives by this time and the weight now is becomes like eight times longer.
126. 12:32So the medium stay as uh the the time that we can satisfy it.
127. 12:38But we we cannot only look at the median time.
128. 12:42So it changed everything that if we only like look at that one.
129. 12:47So the users feel the weight and even when the throughput you see like oh we have through like more than 300 per second but still you need to care about the final weight time.
130. 12:58So the first token may be reasoning and we will still need to measure when the usable report arrives.
131. 13:05So set a dial line for that report.
132. 13:07we choose a low that miss it and there's a one more setting that change the work we ask the model to do.
133. 13:13So this is the first time we uh just use this different uh reasoning uh uh control.
134. 13:21So uh from this part uh so this is the default max changes that we use in the request and it can controls how many tokens we use for this model thinking.
135. 13:35So in the document setup here so thinking stays on but if we leave the effort blank the default is max.
136. 13:43So a default can change the amount of work we request and one coding test there shows the tradeoff.
137. 13:50So high reasoning and max reasoning.
138. 13:53So the high uses fewer up output tokens than the max but the accuracy actually fell from 80% to 68%.
139. 14:04So the fewer tokens came with lower accuracy on those task items.
140. 14:09So give this task the thinking budget it really needs.
141. 14:13So for our research job compare these settings on the same reports and check whether each setting finds the evidence and supports the calculations and the quick first pass on the report ready for review
142. 14:27may need different budgets.
143. 14:29So we need to record this setting or which reasoning control we need to put and then used in our later work.
144. 14:41Okay.
145. 14:42Then when we ship this research workflow with all these um new updates we have with this model.
146. 14:51First we can fix this question on the source reports and use the same information cutoff and agree on what makes a report acceptable before running this comparison and then we can check three things.
147. 15:06The first is can the analysts trust the report and open the resources behind the important claims and check the reporting periods recalculate the financial financial numbers and look for the evidence that could change the conclusion.
148. 15:20And second, what did the finished job cost?
149. 15:24How long did they take?
150. 15:26And include retrieval tool costs and retries.
151. 15:30So if a person must fix the report before use, count that work too.
152. 15:35So include all the failed attempts in the cost of producing uh of producing this acceptable reports.
153. 15:44So test are the number of these users you expect and check whether reports arrive within the promised time and measure the cost of report you can actually use.
154. 15:55And the third one is can we repeat this result save this model and runtime versions and also save the prompt settings and sources.
155. 16:04So keep these examples of failures and we can start with a small roll out and keep a way to switch back.
156. 16:12So before we finish, here's a short recap of today's talk.
157. 16:17It's created with Miniax A3, our AI video model, and you can check here.
158. 16:38Okay.
159. 16:38So uh this is another model that we are working on and it's opened way.
160. 16:43So everyone that who wants to build this actually it's only a one problem with the text to video.
161. 16:49Okay.
162. 16:50So let's return to this uh our opening question today.
163. 16:54Should we switch to a faster model and look at this finish job that we talking about for the last 18 minutes.
164. 17:02So for our analyst that means a report with resources they can open and it has numbers they can check it must arrive when they need it and for uh this M3.1
165. 17:14flash changes how we spend computation and the complete task uh shows that those changes means for the user so measure what we cost to finish job and on time.
166. 17:27So faster models uh I think like faster mo models really matter but what matter most is what we can do with them.
167. 17:35Thank you guys.