WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=YvQ4BVTWfHI NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/YvQ4BVTWfHI.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:12.719 --> 00:00:14.480 Uh I'm Morgan from Minimax. s2 00:00:14.480 --> 00:00:17.680 Uh how many of you knows Minax before? s3 00:00:17.680 --> 00:00:18.640 Oh wow. s4 00:00:18.640 --> 00:00:20.160 I like that. s5 00:00:20.160 --> 00:00:20.480 Yeah. s6 00:00:20.480 --> 00:00:24.720 Because uh I joined this M uh company for three almost three years. s7 00:00:24.720 --> 00:00:30.560 But when we first like step into the American market like uh almost no one knows us. s8 00:00:30.560 --> 00:00:33.440 But at that time we have the first video generation model. s9 00:00:33.440 --> 00:00:36.960 So I will show you something of that at the end of this speech. s10 00:00:36.960 --> 00:00:43.200 But first I will show you like about our the latest language model which will be released next week. s11 00:00:43.200 --> 00:00:49.520 So uh today um I would like to my topic is the hidden cost of a faster model. s12 00:00:49.520 --> 00:00:58.160 So when people thinking about like if you have a AI application your product built in in your own pro workflow. s13 00:00:58.160 --> 00:01:06.159 So when a new model arrives when I say oh please we have a new model do you want to try it? s14 00:01:06.159 --> 00:01:13.680 It can generate a task faster but your I think your question is should I switch to that model? s15 00:01:13.680 --> 00:01:22.240 So you try a research task and the report just comes back very soon and but it misses a very important source. s16 00:01:22.240 --> 00:01:25.360 So someone has to find that source and fix that report. s17 00:01:25.360 --> 00:01:29.040 I think this is the thing we did every every day. s18 00:01:29.040 --> 00:01:35.040 So did the job really get faster with this faster model. s19 00:01:35.600 --> 00:01:44.720 Today we will look at how we reduce the model computation and then we will ask how to test the value of those changes of this model. s20 00:01:44.720 --> 00:01:50.079 So let me show you the job that we will use to answer this question. s21 00:01:50.079 --> 00:02:01.920 So at minimax we build different multimodel models and pro products that people use and a quick chat reply and research report actually have different needs. s22 00:02:01.920 --> 00:02:05.680 So the task tells us what good performance means. s23 00:02:05.680 --> 00:02:12.560 Today I will focus on our latest uh large language model M3.1 flash and one research job. s24 00:02:12.560 --> 00:02:25.360 So here's the analyst question as AI server demand is growing are the companies supplying the circuit boards turning this growth into cash? s25 00:02:25.360 --> 00:02:28.000 The PCB there means printed circuit board. s26 00:02:28.000 --> 00:02:29.599 I think you guys know that. s27 00:02:29.599 --> 00:02:36.720 and unc it connects electronic components that inside devices that such as the AI servers. s28 00:02:36.720 --> 00:02:40.720 So our research case looks like companies in this supply chain. s29 00:02:40.720 --> 00:02:47.519 The internal review describes work across 14 company reports and the model has several jobs. s30 00:02:47.519 --> 00:02:49.440 It must find the right reports. s31 00:02:49.440 --> 00:02:55.280 It must compile the same reporting periods and it must keep different businesses separate. s32 00:02:55.280 --> 00:02:58.959 Then it must explain what the numbers mean. s33 00:02:58.959 --> 00:03:02.560 Think about the analyst who receives the report. s34 00:03:02.560 --> 00:03:08.000 They need to open the source behind an important claim and they need to check a calculation. s35 00:03:08.000 --> 00:03:11.840 They need to see what evidence could change the conclusion. s36 00:03:11.840 --> 00:03:20.800 So a useful report gives the analyst something that they can check and that is our standard for a finished research job. s37 00:03:20.800 --> 00:03:25.680 So when funding shows why those checks matter. s38 00:03:26.400 --> 00:03:30.159 This internal review highlighted a very useful distinction. s39 00:03:30.159 --> 00:03:36.560 So profit can grow but cash flow tells a different story. s40 00:03:36.560 --> 00:03:48.560 A company sells more so is reported pro profit rises but customers have not paid yet or it has uh bought materials that are still in storage. s41 00:03:48.560 --> 00:03:57.599 So the anal the analyst needs to know why the model should point to the cash flow statement and the relevant is disclosures. s42 00:03:57.599 --> 00:04:05.599 So a report earns trust when it claims can be checked and the internal review here describes the research output. s43 00:04:05.599 --> 00:04:12.959 So we still need to verify it sources and calculations and also we also need to measure the time and cost on this same job. s44 00:04:12.959 --> 00:04:17.280 But the technical task a heart use separate tasks. s45 00:04:17.280 --> 00:04:21.199 So they help us to understand specific trade-offs. s46 00:04:21.199 --> 00:04:27.600 With this job clear, so let's look at what changes inside M3.1 flash. s47 00:04:28.639 --> 00:04:33.360 This table shows the main changes inside M3.1 flash. s48 00:04:33.360 --> 00:04:36.800 So I will connect them to a few simple questions. s49 00:04:36.800 --> 00:04:40.960 First, where can we reduce the attention work? s50 00:04:40.960 --> 00:04:45.919 And second, where can we store and move data? s51 00:04:45.919 --> 00:04:52.479 And third, how does the serving setup affect the time a user waits? s52 00:04:52.479 --> 00:04:57.840 And fourth, we also control how much reasoning the model does. s53 00:04:57.840 --> 00:05:04.800 These choices affect different parts of the job and each knows a check against the result we want. s54 00:05:04.800 --> 00:05:09.039 So every optimization has to help the finished job. s55 00:05:09.039 --> 00:05:14.000 And let's begin with how the model uses a long input. s56 00:05:15.120 --> 00:05:22.240 So nowadays I think every model says oh I have a very big context window like we said we have 1 million. s57 00:05:22.240 --> 00:05:30.240 So among this one million how it can help you to find the output find your result quickly faster and correct. s58 00:05:31.039 --> 00:05:39.440 So some passenger passages like in this report like miter just much more than others for this question. s59 00:05:39.440 --> 00:05:48.479 So sparse attention itself means like we uses a very small selection step first like the blue uh box there. s60 00:05:48.479 --> 00:06:01.039 So it scores the blocks of the tax and the main attention then calculation calculate the uses of the selected blocks and the model also keeps the nearby tax in view s61 00:06:01.039 --> 00:06:14.319 and this selection actually they can change with the query and different questions can also needs different passengers of the the context and the number on at the bottom show how much text is published s62 00:06:14.319 --> 00:06:18.800 examples lacks and the full input can be much longer. s63 00:06:18.800 --> 00:06:26.000 For our research job, the goal is to reduce this attention work across many of uh these documents. s64 00:06:26.000 --> 00:06:33.840 But just within this document, a very short footnote there may change the entire conclusion. s65 00:06:33.840 --> 00:06:40.160 So for a cash flow question, the footnote may matter more than the pages of industry outlook. s66 00:06:40.160 --> 00:06:44.240 So we want the model to include that passage too. s67 00:06:44.240 --> 00:06:48.560 less attention work still has to find the important evidence. s68 00:06:48.560 --> 00:06:50.880 So the next question is simple. s69 00:06:50.880 --> 00:07:05.360 Can we still find the facts we need M3.1 flash also uses like the this bar's attention in its first three layers that makes this retrieval an important check. s70 00:07:05.360 --> 00:07:10.960 So this separate task places at different OP positions in long inputs. s71 00:07:10.960 --> 00:07:22.000 So in v run the model passes all 39 requests and the longest inputs were about uh 830k uh tokens. s72 00:07:22.000 --> 00:07:29.440 So the result covers the cases in this test and for our research job we need with real reports. s73 00:07:29.440 --> 00:07:32.960 Can the model find a cash flow footnote? s74 00:07:32.960 --> 00:07:35.440 Does it use the right reporting period? s75 00:07:35.440 --> 00:07:39.680 And can it spot a conflict between different documents? s76 00:07:39.680 --> 00:07:46.639 So move this important passage to different places in the input and then ask the same question again. s77 00:07:46.639 --> 00:07:50.720 A long input is useful when the model can find a key fact. s78 00:07:50.720 --> 00:07:53.360 So finding the evidence is one track too. s79 00:07:53.360 --> 00:07:58.560 Next we look at the numbers we used inside this model. s80 00:07:58.960 --> 00:08:05.120 So the model keeps the internal information from earlier text which we call this KV cache. s81 00:08:05.120 --> 00:08:14.160 And M3.1 flash changes its keys and values from eight bits to four bits, but the query still stays at eight bits. s82 00:08:14.160 --> 00:08:17.840 And this can reduce the data we store and move. s83 00:08:17.840 --> 00:08:21.759 It also changes the position of that internal information. s84 00:08:21.759 --> 00:08:23.039 Look at that table. s85 00:08:23.039 --> 00:08:31.759 So the average scores actually is almost the same but the sign score there it drops like 6.6 percentage. s86 00:08:31.759 --> 00:08:35.519 So some but some other scores rises. s87 00:08:35.519 --> 00:08:42.159 So the average score actually can hide a mistake we your user cares about. s88 00:08:42.159 --> 00:08:48.560 This compares that two full setups but it does not tell us which changes caused each score. s89 00:08:48.560 --> 00:08:53.279 So for our research job we should check the output out output again. s90 00:08:53.279 --> 00:08:54.720 Are the sources right? s91 00:08:54.720 --> 00:08:56.240 Are the calculations right? s92 00:08:56.240 --> 00:08:59.680 And does the explanation match the evidence? s93 00:08:59.680 --> 00:09:05.360 So the same ideas applies to another part of the model is experts. s94 00:09:05.360 --> 00:09:09.760 Uh our model is the MO ine architecture. s95 00:09:09.760 --> 00:09:17.600 So it has the computing parts that we called experts which is the uh M of E experts. s96 00:09:17.600 --> 00:09:22.720 So it selects some of them for each token in M3.1 flash. s97 00:09:22.720 --> 00:09:28.320 These selected experts use four bit ways and act activations. s98 00:09:28.320 --> 00:09:43.120 The shared expert is outside this change and ways of stored model uh values and activations are values the model calculates and lower precision can reduce data movement and support more efficient computation. s99 00:09:43.120 --> 00:09:46.800 But the full calculation has several steps. s100 00:09:46.800 --> 00:09:51.600 So we scale the numbers, we run the calculation and restore the scale. s101 00:09:51.600 --> 00:10:02.720 The position of these steps really matters and the table shows like where scale restore happens and getting that order right is the part of getting the answer right. s102 00:10:02.720 --> 00:10:06.320 So fast math still has to be corrected. s103 00:10:06.320 --> 00:10:11.760 And when we check the complete result, we we measure the complete imp uh implementation. s104 00:10:11.760 --> 00:10:15.360 And for this research job, we run the same questions again. s105 00:10:15.360 --> 00:10:19.920 We inspect why the answers still miss our checks. s106 00:10:19.920 --> 00:10:22.160 And now we look at insight. s107 00:10:22.160 --> 00:10:24.640 We have already look at inside the model. s108 00:10:24.640 --> 00:10:30.320 And now we can look at how it generates an answer for you. s109 00:10:30.800 --> 00:10:36.480 So a serving method called speculative decoding works in two stages. s110 00:10:36.480 --> 00:10:39.360 A faster draft process. s111 00:10:39.360 --> 00:10:54.079 It can help you to propose a short sequence of tokens and the then the targeting model can tracks it and the system keeps this first part that the target accepts and makes a correction when it's needed s112 00:10:54.079 --> 00:10:57.600 and then continues this whole process. s113 00:10:57.600 --> 00:11:04.160 And sometimes it keeps several proposed tokens and sometimes the draft does work what it cannot keep. s114 00:11:04.160 --> 00:11:12.079 So the opportunity here is to get several tokens from one target check and but drafting takes time checking takes time. s115 00:11:12.079 --> 00:11:14.959 Rejecting this tokens also takes time. s116 00:11:14.959 --> 00:11:18.959 So count the time spent checking the draft too. s117 00:11:18.959 --> 00:11:25.600 These checks propose tokens and but financial claims and sources need their own checks. s118 00:11:25.600 --> 00:11:32.320 So for our research job we should also include the source retrieval the tool calls and retries. s119 00:11:32.320 --> 00:11:39.440 The final measure is the time to a report that an analyst can really use in their home own work. s120 00:11:39.440 --> 00:11:47.279 So even when one request runs well many users can change this diff experience. s121 00:11:47.839 --> 00:11:50.320 Here's an separate serving test. s122 00:11:50.320 --> 00:12:01.360 So it's uses the storage generation with low reasoning effort but the concurrence requests now like we rise from the 32s to 64. s123 00:12:01.360 --> 00:12:15.200 So the total output per second grows about 1.9 times but look at the weight time actually it the P95 weight for the first token rises. s124 00:12:15.200 --> 00:12:22.800 It goes from about 1.5 seconds to 11.3 seconds uh6 seconds. s125 00:12:22.800 --> 00:12:32.720 So for 95% of this request the first token arrives by this time and the weight now is becomes like eight times longer. s126 00:12:32.720 --> 00:12:38.160 So the medium stay as uh the the time that we can satisfy it. s127 00:12:38.160 --> 00:12:42.160 But we we cannot only look at the median time. s128 00:12:42.160 --> 00:12:47.519 So it changed everything that if we only like look at that one. s129 00:12:47.519 --> 00:12:58.160 So the users feel the weight and even when the throughput you see like oh we have through like more than 300 per second but still you need to care about the final weight time. s130 00:12:58.160 --> 00:13:05.040 So the first token may be reasoning and we will still need to measure when the usable report arrives. s131 00:13:05.040 --> 00:13:07.360 So set a dial line for that report. s132 00:13:07.360 --> 00:13:13.839 we choose a low that miss it and there's a one more setting that change the work we ask the model to do. s133 00:13:13.839 --> 00:13:21.839 So this is the first time we uh just use this different uh reasoning uh uh control. s134 00:13:21.839 --> 00:13:35.680 So uh from this part uh so this is the default max changes that we use in the request and it can controls how many tokens we use for this model thinking. s135 00:13:35.680 --> 00:13:43.279 So in the document setup here so thinking stays on but if we leave the effort blank the default is max. s136 00:13:43.279 --> 00:13:50.480 So a default can change the amount of work we request and one coding test there shows the tradeoff. s137 00:13:50.480 --> 00:13:53.760 So high reasoning and max reasoning. s138 00:13:53.760 --> 00:14:04.079 So the high uses fewer up output tokens than the max but the accuracy actually fell from 80% to 68%. s139 00:14:04.800 --> 00:14:09.600 So the fewer tokens came with lower accuracy on those task items. s140 00:14:09.600 --> 00:14:13.680 So give this task the thinking budget it really needs. s141 00:14:13.680 --> 00:14:27.600 So for our research job compare these settings on the same reports and check whether each setting finds the evidence and supports the calculations and the quick first pass on the report ready for review s142 00:14:27.600 --> 00:14:29.360 may need different budgets. s143 00:14:29.360 --> 00:14:39.279 So we need to record this setting or which reasoning control we need to put and then used in our later work. s144 00:14:41.680 --> 00:14:42.240 Okay. s145 00:14:42.240 --> 00:14:51.920 Then when we ship this research workflow with all these um new updates we have with this model. s146 00:14:51.920 --> 00:15:06.480 First we can fix this question on the source reports and use the same information cutoff and agree on what makes a report acceptable before running this comparison and then we can check three things. s147 00:15:06.480 --> 00:15:20.880 The first is can the analysts trust the report and open the resources behind the important claims and check the reporting periods recalculate the financial financial numbers and look for the evidence that could change the conclusion. s148 00:15:20.880 --> 00:15:24.480 And second, what did the finished job cost? s149 00:15:24.480 --> 00:15:26.079 How long did they take? s150 00:15:26.079 --> 00:15:30.320 And include retrieval tool costs and retries. s151 00:15:30.320 --> 00:15:35.279 So if a person must fix the report before use, count that work too. s152 00:15:35.279 --> 00:15:43.760 So include all the failed attempts in the cost of producing uh of producing this acceptable reports. s153 00:15:44.639 --> 00:15:55.120 So test are the number of these users you expect and check whether reports arrive within the promised time and measure the cost of report you can actually use. s154 00:15:55.120 --> 00:16:04.399 And the third one is can we repeat this result save this model and runtime versions and also save the prompt settings and sources. s155 00:16:04.399 --> 00:16:12.480 So keep these examples of failures and we can start with a small roll out and keep a way to switch back. s156 00:16:12.720 --> 00:16:17.759 So before we finish, here's a short recap of today's talk. s157 00:16:17.759 --> 00:16:24.720 It's created with Miniax A3, our AI video model, and you can check here. s158 00:16:38.240 --> 00:16:38.800 Okay. s159 00:16:38.800 --> 00:16:43.680 So uh this is another model that we are working on and it's opened way. s160 00:16:43.680 --> 00:16:49.440 So everyone that who wants to build this actually it's only a one problem with the text to video. s161 00:16:49.440 --> 00:16:50.399 Okay. s162 00:16:50.399 --> 00:16:54.399 So let's return to this uh our opening question today. s163 00:16:54.399 --> 00:17:02.160 Should we switch to a faster model and look at this finish job that we talking about for the last 18 minutes. s164 00:17:02.160 --> 00:17:14.000 So for our analyst that means a report with resources they can open and it has numbers they can check it must arrive when they need it and for uh this M3.1 s165 00:17:14.000 --> 00:17:27.520 flash changes how we spend computation and the complete task uh shows that those changes means for the user so measure what we cost to finish job and on time. s166 00:17:27.520 --> 00:17:35.760 So faster models uh I think like faster mo models really matter but what matter most is what we can do with them. s167 00:17:35.760 --> 00:17:38.160 Thank you guys.