Is a Faster Model Actually Faster? Test the Whole Job — Morgan Suo, MiniMax
https://www.youtube.com/watch?v=YvQ4BVTWfHI
AI Engineer · 17 min · 167 sentences · sentence-level transcript from YouTube's caption track; each line begins at the time shown

[00:12] Uh I'm Morgan from Minimax.
[00:14] Uh how many of you knows Minax before?
[00:17] Oh wow.
[00:18] I like that.
[00:20] Yeah.
[00:20] Because uh I joined this M uh company for three almost three years.
[00:24] But when we first like step into the American market like uh almost no one knows us.
[00:30] But at that time we have the first video generation model.
[00:33] So I will show you something of that at the end of this speech.
[00:36] But first I will show you like about our the latest language model which will be released next week.
[00:43] So uh today um I would like to my topic is the hidden cost of a faster model.
[00:49] So when people thinking about like if you have a AI application your product built in in your own pro workflow.
[00:58] So when a new model arrives when I say oh please we have a new model do you want to try it?
[01:06] It can generate a task faster but your I think your question is should I switch to that model?
[01:13] So you try a research task and the report just comes back very soon and but it misses a very important source.
[01:22] So someone has to find that source and fix that report.
[01:25] I think this is the thing we did every every day.
[01:29] So did the job really get faster with this faster model.
[01:35] Today we will look at how we reduce the model computation and then we will ask how to test the value of those changes of this model.
[01:44] So let me show you the job that we will use to answer this question.
[01:50] So at minimax we build different multimodel models and pro products that people use and a quick chat reply and research report actually have different needs.
[02:01] So the task tells us what good performance means.
[02:05] Today I will focus on our latest uh large language model M3.1 flash and one research job.
[02:12] So here's the analyst question as AI server demand is growing are the companies supplying the circuit boards turning this growth into cash?
[02:25] The PCB there means printed circuit board.
[02:28] I think you guys know that.
[02:29] and unc it connects electronic components that inside devices that such as the AI servers.
[02:36] So our research case looks like companies in this supply chain.
[02:40] The internal review describes work across 14 company reports and the model has several jobs.
[02:47] It must find the right reports.
[02:49] It must compile the same reporting periods and it must keep different businesses separate.
[02:55] Then it must explain what the numbers mean.
[02:58] Think about the analyst who receives the report.
[03:02] They need to open the source behind an important claim and they need to check a calculation.
[03:08] They need to see what evidence could change the conclusion.
[03:11] So a useful report gives the analyst something that they can check and that is our standard for a finished research job.
[03:20] So when funding shows why those checks matter.
[03:26] This internal review highlighted a very useful distinction.
[03:30] So profit can grow but cash flow tells a different story.
[03:36] A company sells more so is reported pro profit rises but customers have not paid yet or it has uh bought materials that are still in storage.
[03:48] So the anal the analyst needs to know why the model should point to the cash flow statement and the relevant is disclosures.
[03:57] So a report earns trust when it claims can be checked and the internal review here describes the research output.
[04:05] So we still need to verify it sources and calculations and also we also need to measure the time and cost on this same job.
[04:12] But the technical task a heart use separate tasks.
[04:17] So they help us to understand specific trade-offs.
[04:21] With this job clear, so let's look at what changes inside M3.1 flash.
[04:28] This table shows the main changes inside M3.1 flash.
[04:33] So I will connect them to a few simple questions.
[04:36] First, where can we reduce the attention work?
[04:40] And second, where can we store and move data?
[04:45] And third, how does the serving setup affect the time a user waits?
[04:52] And fourth, we also control how much reasoning the model does.
[04:57] These choices affect different parts of the job and each knows a check against the result we want.
[05:04] So every optimization has to help the finished job.
[05:09] And let's begin with how the model uses a long input.
[05:15] So nowadays I think every model says oh I have a very big context window like we said we have 1 million.
[05:22] So among this one million how it can help you to find the output find your result quickly faster and correct.
[05:31] So some passenger passages like in this report like miter just much more than others for this question.
[05:39] So sparse attention itself means like we uses a very small selection step first like the blue uh box there.
[05:48] So it scores the blocks of the tax and the main attention then calculation calculate the uses of the selected blocks and the model also keeps the nearby tax in view
[06:01] and this selection actually they can change with the query and different questions can also needs different passengers of the the context and the number on at the bottom show how much text is published
[06:14] examples lacks and the full input can be much longer.
[06:18] For our research job, the goal is to reduce this attention work across many of uh these documents.
[06:26] But just within this document, a very short footnote there may change the entire conclusion.
[06:33] So for a cash flow question, the footnote may matter more than the pages of industry outlook.
[06:40] So we want the model to include that passage too.
[06:44] less attention work still has to find the important evidence.
[06:48] So the next question is simple.
[06:50] Can we still find the facts we need M3.1 flash also uses like the this bar's attention in its first three layers that makes this retrieval an important check.
[07:05] So this separate task places at different OP positions in long inputs.
[07:10] So in v run the model passes all 39 requests and the longest inputs were about uh 830k uh tokens.
[07:22] So the result covers the cases in this test and for our research job we need with real reports.
[07:29] Can the model find a cash flow footnote?
[07:32] Does it use the right reporting period?
[07:35] And can it spot a conflict between different documents?
[07:39] So move this important passage to different places in the input and then ask the same question again.
[07:46] A long input is useful when the model can find a key fact.
[07:50] So finding the evidence is one track too.
[07:53] Next we look at the numbers we used inside this model.
[07:58] So the model keeps the internal information from earlier text which we call this KV cache.
[08:05] And M3.1 flash changes its keys and values from eight bits to four bits, but the query still stays at eight bits.
[08:14] And this can reduce the data we store and move.
[08:17] It also changes the position of that internal information.
[08:21] Look at that table.
[08:23] So the average scores actually is almost the same but the sign score there it drops like 6.6 percentage.
[08:31] So some but some other scores rises.
[08:35] So the average score actually can hide a mistake we your user cares about.
[08:42] This compares that two full setups but it does not tell us which changes caused each score.
[08:48] So for our research job we should check the output out output again.
[08:53] Are the sources right?
[08:54] Are the calculations right?
[08:56] And does the explanation match the evidence?
[08:59] So the same ideas applies to another part of the model is experts.
[09:05] Uh our model is the MO ine architecture.
[09:09] So it has the computing parts that we called experts which is the uh M of E experts.
[09:17] So it selects some of them for each token in M3.1 flash.
[09:22] These selected experts use four bit ways and act activations.
[09:28] The shared expert is outside this change and ways of stored model uh values and activations are values the model calculates and lower precision can reduce data movement and support more efficient computation.
[09:43] But the full calculation has several steps.
[09:46] So we scale the numbers, we run the calculation and restore the scale.
[09:51] The position of these steps really matters and the table shows like where scale restore happens and getting that order right is the part of getting the answer right.
[10:02] So fast math still has to be corrected.
[10:06] And when we check the complete result, we we measure the complete imp uh implementation.
[10:11] And for this research job, we run the same questions again.
[10:15] We inspect why the answers still miss our checks.
[10:19] And now we look at insight.
[10:22] We have already look at inside the model.
[10:24] And now we can look at how it generates an answer for you.
[10:30] So a serving method called speculative decoding works in two stages.
[10:36] A faster draft process.
[10:39] It can help you to propose a short sequence of tokens and the then the targeting model can tracks it and the system keeps this first part that the target accepts and makes a correction when it's needed
[10:54] and then continues this whole process.
[10:57] And sometimes it keeps several proposed tokens and sometimes the draft does work what it cannot keep.
[11:04] So the opportunity here is to get several tokens from one target check and but drafting takes time checking takes time.
[11:12] Rejecting this tokens also takes time.
[11:14] So count the time spent checking the draft too.
[11:18] These checks propose tokens and but financial claims and sources need their own checks.
[11:25] So for our research job we should also include the source retrieval the tool calls and retries.
[11:32] The final measure is the time to a report that an analyst can really use in their home own work.
[11:39] So even when one request runs well many users can change this diff experience.
[11:47] Here's an separate serving test.
[11:50] So it's uses the storage generation with low reasoning effort but the concurrence requests now like we rise from the 32s to 64.
[12:01] So the total output per second grows about 1.9 times but look at the weight time actually it the P95 weight for the first token rises.
[12:15] It goes from about 1.5 seconds to 11.3 seconds uh6 seconds.
[12:22] So for 95% of this request the first token arrives by this time and the weight now is becomes like eight times longer.
[12:32] So the medium stay as uh the the time that we can satisfy it.
[12:38] But we we cannot only look at the median time.
[12:42] So it changed everything that if we only like look at that one.
[12:47] So the users feel the weight and even when the throughput you see like oh we have through like more than 300 per second but still you need to care about the final weight time.
[12:58] So the first token may be reasoning and we will still need to measure when the usable report arrives.
[13:05] So set a dial line for that report.
[13:07] we choose a low that miss it and there's a one more setting that change the work we ask the model to do.
[13:13] So this is the first time we uh just use this different uh reasoning uh uh control.
[13:21] So uh from this part uh so this is the default max changes that we use in the request and it can controls how many tokens we use for this model thinking.
[13:35] So in the document setup here so thinking stays on but if we leave the effort blank the default is max.
[13:43] So a default can change the amount of work we request and one coding test there shows the tradeoff.
[13:50] So high reasoning and max reasoning.
[13:53] So the high uses fewer up output tokens than the max but the accuracy actually fell from 80% to 68%.
[14:04] So the fewer tokens came with lower accuracy on those task items.
[14:09] So give this task the thinking budget it really needs.
[14:13] So for our research job compare these settings on the same reports and check whether each setting finds the evidence and supports the calculations and the quick first pass on the report ready for review
[14:27] may need different budgets.
[14:29] So we need to record this setting or which reasoning control we need to put and then used in our later work.
[14:41] Okay.
[14:42] Then when we ship this research workflow with all these um new updates we have with this model.
[14:51] First we can fix this question on the source reports and use the same information cutoff and agree on what makes a report acceptable before running this comparison and then we can check three things.
[15:06] The first is can the analysts trust the report and open the resources behind the important claims and check the reporting periods recalculate the financial financial numbers and look for the evidence that could change the conclusion.
[15:20] And second, what did the finished job cost?
[15:24] How long did they take?
[15:26] And include retrieval tool costs and retries.
[15:30] So if a person must fix the report before use, count that work too.
[15:35] So include all the failed attempts in the cost of producing uh of producing this acceptable reports.
[15:44] So test are the number of these users you expect and check whether reports arrive within the promised time and measure the cost of report you can actually use.
[15:55] And the third one is can we repeat this result save this model and runtime versions and also save the prompt settings and sources.
[16:04] So keep these examples of failures and we can start with a small roll out and keep a way to switch back.
[16:12] So before we finish, here's a short recap of today's talk.
[16:17] It's created with Miniax A3, our AI video model, and you can check here.
[16:38] Okay.
[16:38] So uh this is another model that we are working on and it's opened way.
[16:43] So everyone that who wants to build this actually it's only a one problem with the text to video.
[16:49] Okay.
[16:50] So let's return to this uh our opening question today.
[16:54] Should we switch to a faster model and look at this finish job that we talking about for the last 18 minutes.
[17:02] So for our analyst that means a report with resources they can open and it has numbers they can check it must arrive when they need it and for uh this M3.1
[17:14] flash changes how we spend computation and the complete task uh shows that those changes means for the user so measure what we cost to finish job and on time.
[17:27] So faster models uh I think like faster mo models really matter but what matter most is what we can do with them.
[17:35] Thank you guys.