WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=Ti-BmGU1_mQ NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/Ti-BmGU1_mQ.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:01.309 --> 00:00:03.309 [music] s2 00:00:12.640 --> 00:00:14.520 Uh thank you for everyone for coming here. s3 00:00:14.520 --> 00:00:18.560 Uh not too sure whether to be happy or sad that so many people are suffering from the token bill. s4 00:00:18.560 --> 00:00:20.120 It's a full workshop. s5 00:00:20.120 --> 00:00:24.240 Uh Yeah, that uh we're going to talk about tokenomics. s6 00:00:24.240 --> 00:00:27.440 So, a little bit of a a background with the kickstarting it, right? s7 00:00:27.440 --> 00:00:28.920 I'm Eugene from Felis AI. s8 00:00:28.920 --> 00:00:30.840 I'm CEO and founder of of it. s9 00:00:30.840 --> 00:00:36.320 Uh and I'm part of the team behind the Arabic AI open source project, the first AI model under the Linux Foundation. s10 00:00:36.320 --> 00:00:39.120 And and we work on next generation AI architecture. s11 00:00:39.120 --> 00:00:41.160 Uh that's a little bit of my academic background. s12 00:00:41.160 --> 00:00:43.320 And my Twitter handle is PicoCreator. s13 00:00:43.320 --> 00:00:45.840 But, but more importantly, right? s14 00:00:45.840 --> 00:00:54.680 Uh uh a little bit of background for Felis AI is that our mission is to make AI accessible for everyone, every language, every dialect, every country. s15 00:00:54.680 --> 00:01:01.240 And we do this We do this by creating the world's largest platform of open models, of over 40,000 open models. s16 00:01:01.240 --> 00:01:09.120 And this includes specialized fine-tune like the Sunflower model, which is Uganda's first language model, specifically for their language, and a lot of other specialized models. s17 00:01:09.120 --> 00:01:17.400 So, we're not just not talking about the biggest open source model, but specialized language models or specialized use cases, uh some of it which I'll touch as well. s18 00:01:17.400 --> 00:01:24.640 And when we when we can go on scale in terms of variety of model, Felis AI also we serve customers at scale. s19 00:01:24.640 --> 00:01:29.120 So, some of our largest customers are literally serving 2 trillion tokens a day. s20 00:01:29.120 --> 00:01:35.160 And and they're And and they're serving it at around 0.3 per million tokens or 60K a day. s21 00:01:35.160 --> 00:01:39.640 And they're doing it on around 512 AMD GPUs, uh MI215. s22 00:01:39.640 --> 00:01:44.560 Uh and for this customer in particular, it's a private 400 Android BM OE model. s23 00:01:44.560 --> 00:01:48.400 Uh and they're serving it around Opus 4.8 class performance. s24 00:01:48.400 --> 00:01:53.320 So, it's lots of concurrent requests, lots of token volume. s25 00:01:53.320 --> 00:01:55.160 That's what we mean by scale. s26 00:01:55.160 --> 00:02:04.440 And if you look at it, 0.03 per million token at over 4.8 class, it's not something that is not something that you can get off the public cloud market. s27 00:02:04.440 --> 00:02:09.399 This is specially very specialized setups that is meant to run at the scale that we are talking about. s28 00:02:09.399 --> 00:02:15.400 So, that kind of uh leads in into two major disclaimers that I'm going to start off. s29 00:02:15.400 --> 00:02:17.560 One, we sell inference. s30 00:02:17.560 --> 00:02:22.200 So, everything I cover here will cover the will will touch on any provider. s31 00:02:22.200 --> 00:02:25.000 Nothing here is specific to Fiddler's. s32 00:02:25.000 --> 00:02:27.720 But, to to for clarity, we sell inference. s33 00:02:27.720 --> 00:02:32.640 So, so this is from our data points, from our experience, respectively, as well. s34 00:02:32.640 --> 00:02:33.880 Okay? s35 00:02:33.880 --> 00:02:44.840 The other thing is that this workshop is going to cover a lot of optimization at scale, and it probably will not fit you if you're like spending less than 10K a month in your organization. s36 00:02:44.840 --> 00:02:53.320 Uh my word of advice for those who are below this this uh metric mark, and this is an arbitrary metric, is that if you're spending less than this, you're probably s37 00:02:53.320 --> 00:03:00.680 better off focusing on like trying to get strong use cases within your organization, rather than trying to optimize the token spending bill. s38 00:03:00.680 --> 00:03:05.120 Because a strong use case can bring your organization way more than 10K a month. s39 00:03:05.120 --> 00:03:11.920 And and that's and that's what should be your main area of focus, and that's not my area of focus for this talk. s40 00:03:11.959 --> 00:03:25.760 So, what we are going to cover in this workshop uh is four moves in in in four major order: measuring, right-sizing, uh how do you buy your capacity, and how you check your process. s41 00:03:26.120 --> 00:03:28.320 So, measuring. s42 00:03:28.320 --> 00:03:28.920 Yep. s43 00:03:28.920 --> 00:03:31.040 You'll be surprised at how essential this is. s44 00:03:31.040 --> 00:03:34.480 Uh like, how many of you know your total token spend? s45 00:03:34.480 --> 00:03:37.280 Can I have a raise of hands? s46 00:03:37.280 --> 00:03:39.560 Wow, only half the room raised hand. s47 00:03:39.560 --> 00:03:40.080 Okay. s48 00:03:40.080 --> 00:03:43.080 How many of you can measure at the department level? s49 00:03:43.080 --> 00:03:45.880 Those those who have their hands raised up, leave your up. s50 00:03:45.880 --> 00:03:48.840 And so, I see about another half went down. s51 00:03:48.840 --> 00:03:49.480 Yeah. s52 00:03:49.480 --> 00:03:53.800 And it gets lower and lower down to the team level, to the use case level. s53 00:03:53.800 --> 00:04:03.720 Uh we like easily 30 40% of our customers that reach out to us, the first thing they had was a single shared key that everyone in the organization used. s54 00:04:03.720 --> 00:04:07.440 And that was like Uh uh how do we even measure that? s55 00:04:07.440 --> 00:04:08.920 How do we start from there? s56 00:04:08.920 --> 00:04:20.280 So So this is not and a vendor specific like you need to start splitting up your keys, start splitting up by departments, start splitting up by major use cases so that you know where all your tokens s57 00:04:20.280 --> 00:04:21.120 are. s58 00:04:21.120 --> 00:04:25.520 Um and this is then from then on it can inform your workload. s59 00:04:25.520 --> 00:04:28.320 So yeah, API keys shouldn't be that hard to issue. s60 00:04:28.320 --> 00:04:32.440 Um once you know that, right, you can start right-sizing things uh respectively. s61 00:04:32.440 --> 00:04:35.240 Like like which team is uh and workloads spend on it? s62 00:04:35.240 --> 00:04:36.480 What was the person waiting for? s63 00:04:36.480 --> 00:04:39.480 Like some teams deserve the tokens per second. s64 00:04:39.480 --> 00:04:45.240 Some background agents running at midnight probably can don't deserve the tokens per second speed that you're paying for it. s65 00:04:45.240 --> 00:04:48.760 Uh yeah, don't pay for ultra on a background agent. s66 00:04:48.760 --> 00:04:49.919 That's just dumb. s67 00:04:49.919 --> 00:05:00.120 Um and and from that also you can also look at your usage to better advise like uh dedicated capacity that you may purchase, be it for open models or even close models. s68 00:05:00.120 --> 00:05:01.400 Uh yeah. s69 00:05:01.400 --> 00:05:06.120 And also you can also check which team spend the most token and whether they are productive. s70 00:05:06.120 --> 00:05:09.200 Uh I'll touch on that as well. s71 00:05:09.640 --> 00:05:18.280 One of the things that you should know in AI workloads, right, uh and you all may already experience, right, is there are two major tracks with very opposite economics s72 00:05:18.280 --> 00:05:19.240 uh respectively. s73 00:05:19.240 --> 00:05:24.680 So there is the agentic workload and obviously there's things in between where it tends to be like long-running multitenant agents. s74 00:05:24.680 --> 00:05:27.360 Coding agents are a particular example of this. s75 00:05:27.360 --> 00:05:35.840 Some Some Some may have some devs like run coding agents like literally overnight over a long period of time with uh with multiple compaction. s76 00:05:35.840 --> 00:05:40.760 And and these things maximize the context uh usage. s77 00:05:40.760 --> 00:05:43.880 Process automation is the other extreme. s78 00:05:43.880 --> 00:05:48.960 Uh think call centers, think email routing, things your meeting notes transcribing, and things like that. s79 00:05:48.960 --> 00:05:55.480 It's all It's much shorter in context in general, though there are a lot of examples that defy this statement. s80 00:05:55.480 --> 00:06:03.560 But, it's very process-driven, very task-isolated, and it's it's a volume game, essentially. s81 00:06:04.280 --> 00:06:07.120 So, well, what do I mean by more usage? s82 00:06:07.120 --> 00:06:11.480 And this is probably one of the first major explosion a lot of organizations face. s83 00:06:11.480 --> 00:06:15.280 In particular, when they jump onto the open cloud hype or one of the cloud clones. s84 00:06:15.280 --> 00:06:26.040 So, you like like from from from various developer teams that we we support, like for example, we sampled one of the team, and it was only five devs, and they were averaging around continuously s85 00:06:26.040 --> 00:06:29.160 eight running agents for one of the companies we work with. s86 00:06:29.160 --> 00:06:38.160 And And so, they were previously running Opus 4.8 of the API, and they were essentially spending 150k a month continuously running these agents. s87 00:06:38.160 --> 00:06:41.640 Some of it is the agents that they use to code, some of it is the agents that kick in at night. s88 00:06:41.640 --> 00:06:42.720 They They were slightly more advanced. s89 00:06:42.720 --> 00:06:47.560 They already knew that they they they they should kick things off at night when no one's are working. s90 00:06:47.560 --> 00:06:50.680 But, essentially, they were spending 116 billion tokens a month. s91 00:06:50.680 --> 00:06:55.400 And we literally helped them switch to GRM 5.3 instance. s92 00:06:55.400 --> 00:06:58.960 Previously, it was 5.2. They already upgraded 5.3, and that was like a single node. s93 00:06:58.960 --> 00:07:00.440 That was less than 10k a month. s94 00:07:00.440 --> 00:07:03.320 And they were And they didn't see any disruption or impact. s95 00:07:03.320 --> 00:07:08.640 So, knowing your team allows you to make these decisions respectively. s96 00:07:08.640 --> 00:07:14.120 And like like I highlighted, like knowing processes workload also allows you to split things off. s97 00:07:14.120 --> 00:07:23.600 So, like, for example, previously, they were running everything throughout the day, but they they started scheduling some of the background agents when the team was asleep, and it runs through night. s98 00:07:23.600 --> 00:07:27.919 So, this allows you to utilize their hardware respectively. s99 00:07:28.240 --> 00:07:35.840 The As I I touched on briefly, the other side of process automation is purely a volume game. s100 00:07:35.840 --> 00:07:37.520 Extremely predefined. s101 00:07:37.520 --> 00:07:48.520 I I can give you more specific example off the off the record, but you think of it as like millions per day or even tens of millions per day, decisions in call centers, lines of code scanned, s102 00:07:48.520 --> 00:07:49.880 emails and text messages filtered. s103 00:07:49.880 --> 00:07:56.720 Like, one of the telcos we work with, like, they easily hit peak 10 million messages in a single minute. s104 00:07:56.720 --> 00:07:58.280 And this shouldn't come as a surprise. s105 00:07:58.280 --> 00:07:59.600 Happy New Year is the worst. s106 00:07:59.600 --> 00:07:59.960 Yeah. s107 00:07:59.960 --> 00:08:08.040 So So So these And they And they help And the reason why they filter messages is because they provide a feature about like providing safe phone for kids. s108 00:08:08.040 --> 00:08:19.919 So they needed an AI to scan like basically is it profanity, sexual content, etc. etc. So So this This is really not a long context, but a volume game. s109 00:08:19.919 --> 00:08:25.800 Other things like Other things is like maker-checker processes, approval processes, document processing. s110 00:08:25.800 --> 00:08:28.360 These are all volume game for multiple organizations. s111 00:08:28.360 --> 00:08:36.479 So I suspect for the financial institutions here, since we are in New York, a lot of OCRs, a lot of documents, a lot of approval, a lot of transaction risk s112 00:08:36.479 --> 00:08:39.360 the is is under this category. s113 00:08:39.360 --> 00:08:40.200 Okay. s114 00:08:40.200 --> 00:08:41.479 So yeah. s115 00:08:41.479 --> 00:08:50.480 If you're not yet tracking your token token spend and token view split, right, we are open sourcing a tokenomics tracking tool, which is just a very fancy way of like s116 00:08:50.480 --> 00:08:56.600 splitting up your API key and tracking which API key spends how much, which is separate from this talk. s117 00:08:56.600 --> 00:08:57.040 Yeah. s118 00:08:57.040 --> 00:09:01.760 So if you If you want to again update it, feel free to to scan this QR code and respectively. s119 00:09:01.760 --> 00:09:10.640 It also if you want to slides for this talk and when the recording is released from AI engineering to be notified about this talk, you can also get fill up the form there. s120 00:09:10.640 --> 00:09:11.440 Okay. s121 00:09:11.440 --> 00:09:11.960 Okay. s122 00:09:11.960 --> 00:09:12.920 Okay. s123 00:09:12.920 --> 00:09:14.280 So yeah. s124 00:09:14.280 --> 00:09:16.000 That was about measuring. s125 00:09:16.000 --> 00:09:16.960 Um yeah. s126 00:09:16.960 --> 00:09:25.520 If you are not measuring, you can pretty much ignore everything else I say and start with measuring first because all your biggest ROI is just by measuring. s127 00:09:25.520 --> 00:09:30.680 But once you have things measured, right, the question is how do you right-size it? s128 00:09:30.680 --> 00:09:33.320 And this is where things gets more complicated. s129 00:09:33.320 --> 00:09:47.280 Uh In In general, right, one of the things that we start to observe as as AI gets deployed at scale within organization is that there tends to be a pattern of like because AI is the shiny new hammer, s130 00:09:47.280 --> 00:09:49.440 we try to apply it to everything. s131 00:09:49.440 --> 00:09:53.520 Especially within the innovation labs, we just try to apply to every damn thing. s132 00:09:53.520 --> 00:10:01.080 And we tend and and I have seen organizations literally apply fable to really the silliest of things. s133 00:10:01.080 --> 00:10:13.160 So so you there's a lot of models and a lot of models are really expensive and and and and and figuring out to do and where it fits is is is important s134 00:10:13.160 --> 00:10:16.160 is an important step of the process. s135 00:10:17.240 --> 00:10:21.880 I know open models I mean a lot of you probably heard about open models and that's our line of business. s136 00:10:21.880 --> 00:10:22.880 Yeah. s137 00:10:22.880 --> 00:10:31.040 And and so a lot when a lot of companies come to us for example to like find open models that fit their use cases and drop in replacements. s138 00:10:31.040 --> 00:10:34.040 This is we generally guide them along their use cases. s139 00:10:34.040 --> 00:10:42.440 So these are these are general rule of thumb that is probably going to be updated month because open models release every few months and same thing close source models. s140 00:10:42.440 --> 00:10:43.839 Things change rapidly. s141 00:10:43.839 --> 00:10:50.000 But but there but the thing is what we find right exciting actually within this year is things have started to change. s142 00:10:50.000 --> 00:10:57.000 Like in the previous year the best close source model is without better than the best open source. s143 00:10:57.000 --> 00:10:58.640 Almost. s144 00:10:58.640 --> 00:11:02.920 But as this year approach right we are starting to see the gap close. s145 00:11:02.920 --> 00:11:08.720 We are starting to see situations where where some of the open models are even on par or even surpassing. s146 00:11:08.720 --> 00:11:18.680 So like for example before the Opus 5.5 update came out Kimi K3 was beating Opus in UI design in some of the UI benchmarks. s147 00:11:18.680 --> 00:11:22.600 And and this is like very public benchmarks. s148 00:11:22.600 --> 00:11:29.440 Within customer benchmarks right we were already starting to see open source models beat very specialized benchmarks in specialized use cases. s149 00:11:29.440 --> 00:11:33.920 So like A particular one to highlight is like for example, step one five model. s150 00:11:33.920 --> 00:11:43.320 They it was able to achieve near Opus and extra quality despite being like a 10 x smaller model in financial benchmarks uh financial use cases. s151 00:11:43.320 --> 00:11:45.840 And this is it the best coding model? s152 00:11:45.840 --> 00:11:49.840 No, but in that use case, it's actually extremely strong and competent. s153 00:11:49.840 --> 00:11:58.800 So so so the landscape is slowly changing where where where you can have drop-in replacement for a lot of use cases that are sometimes even just 10 x cheaper s154 00:11:58.800 --> 00:12:04.320 respectively for the uh uh if if you know your workload and you can drop it in respectively. s155 00:12:04.320 --> 00:12:07.360 Special shoutout is also DeepSeek-V4 and Flash. s156 00:12:07.360 --> 00:12:17.120 For some reason, and this model is extremely popular across all workloads respectively because it's easy to prompt there uh for a lot of companies. s157 00:12:17.120 --> 00:12:22.760 So well I well the rest of the models I have listed there they are they are like default for good in their categories. s158 00:12:22.760 --> 00:12:27.120 DeepSeek-V4 and Flash seems to like be quite popular regardless the case. s159 00:12:27.120 --> 00:12:27.840 Yeah. s160 00:12:27.840 --> 00:12:31.280 Uh though benchmark score doesn't seem to justify why why is it so popular. s161 00:12:31.280 --> 00:12:32.160 Yeah. s162 00:12:32.160 --> 00:12:32.880 Yeah. s163 00:12:32.880 --> 00:12:44.920 Uh another thing to highlight, right, is that recently like and this is like for a few weeks ago, uh the a new class of uh AI models have been introduced called uh classifiers s164 00:12:44.920 --> 00:12:48.440 one-shot classifiers most popularly known as JAF. s165 00:12:48.440 --> 00:12:51.839 Open-source projects have already started uh taking off like simple JAF. s166 00:12:51.839 --> 00:12:58.360 And these are models that are entirely focused on just making decisions and classification on on tasks. s167 00:12:58.360 --> 00:13:07.839 Uh the main advantage is that because on a technical level only, they are commonly 10 x cheaper than existing models or even faster at the same time. s168 00:13:07.839 --> 00:13:11.040 Uh the downside is it cannot give a chat response. s169 00:13:11.040 --> 00:13:21.320 So so so these are things that are changing and happening that you all may want to start looking into on like on like segments of how you want to improve your your token economy. s170 00:13:21.320 --> 00:13:23.000 Yeah. s171 00:13:23.000 --> 00:13:35.480 But but back to the The the the token economy part, there is two major tracks and it is important to separate out and track this track them respectively. s172 00:13:35.480 --> 00:13:39.480 It on the agentic side we're talking about long running long context as I said earlier. s173 00:13:39.480 --> 00:13:51.160 And to and you have to understand this workload tends to be extremely cash heavy, extremely input preview heavy and cash heavy and there's only a small percentage of output. s174 00:13:51.160 --> 00:13:57.640 We see across like agentic use use cases in employment only 1% of the request is to the output tokens. s175 00:13:57.640 --> 00:14:03.160 And and and this is this is very logical because it takes multiple turns, rinse and repeat. s176 00:14:03.160 --> 00:14:09.520 But this also decides your token cost because like if you've seen token cost, right? s177 00:14:09.520 --> 00:14:10.320 Deployment cost, right? s178 00:14:10.320 --> 00:14:21.000 Output is may have the biggest number, but even your input value is much smaller or the cash is much smaller, it it gets repeated heavily in in this use case. s179 00:14:21.000 --> 00:14:22.920 Yeah. s180 00:14:22.920 --> 00:14:30.360 This is not coding agent workshop, but one of the most common patterns that we we see in a lot of organizations, right? s181 00:14:30.360 --> 00:14:38.360 Is that is that typically typically in terms of like trying to scale up their AI agent deployments we within within the coding teams, right? s182 00:14:38.360 --> 00:14:48.720 They tend to have the best agent do the planning and a faster a faster cheaper agent do do the implementation and then maybe the best agent do the review again. s183 00:14:48.720 --> 00:15:00.240 There's a good reason why this split is very common across all the agent frameworks is is that you get the best of both worlds because like the implementation phase is tends to be tends to be very token heavy process. s184 00:15:00.240 --> 00:15:06.760 A lot of processes repeated back and forth like key modifying the file, testing, checking for results, rinse and repeating. s185 00:15:06.760 --> 00:15:18.560 And and you and most and most of these agents are especially this especially like Gemini 5.3 flash is more than capable of doing a task at the same quality level s186 00:15:18.560 --> 00:15:25.920 as a much more stronger agent like I say Astra or Opus, uh respectively assuming those agents do the planning. s187 00:15:25.920 --> 00:15:38.720 We have worked with some of the organizations where literally the planning and review agent is the best frontier close muscle model, and then the workers that implement it is the GRM 5.3 flash as an example, or Deep Seek 4.1 flash. s188 00:15:38.720 --> 00:15:40.720 Yeah. s189 00:15:40.720 --> 00:15:53.800 One of the observations that we made from from this from from this kind of split pattern, right, is that unfortunately, as much as we wish that everyone does so, nobody really switches their model to save the company money. s190 00:15:53.800 --> 00:15:59.160 Um this people are lazy to switch the models in general. s191 00:15:59.160 --> 00:16:02.280 You you want to give an incentive to switch. s192 00:16:02.280 --> 00:16:09.280 Or or alternatively, if your agent harness is good enough, so like for example, some harness like M, they automate the process. s193 00:16:09.280 --> 00:16:19.040 Like like it uses a the weaker model as a baseline, and then it uses a big the smarter model to automate switching when it needs a smarter consultation. s194 00:16:19.040 --> 00:16:27.520 This This helps reduce the friction respectively, and helps improve your your token economics at user scale. s195 00:16:27.520 --> 00:16:37.920 Like a a major rule of thumb that we we now give our clients after going through a few times is that you want the smarter model, and since it's already so expensive s196 00:16:37.920 --> 00:16:44.200 as a baseline, but you but you want the the faster model to the cheaper model to be faster. s197 00:16:44.200 --> 00:16:46.800 And that naturally sets an incentive. s198 00:16:46.800 --> 00:16:48.680 People want to get their work done fast. s199 00:16:48.680 --> 00:16:54.960 If the model is smart enough to get it done and it's cheaper, they will just use it because it's faster, not because it's cheaper. s200 00:16:54.960 --> 00:17:06.839 Um we had teams where where where both models were the smarter one and the cheaper one was maintained at the same tokens per second speed, and it and as long as they had the token budget in excess, right, s201 00:17:06.839 --> 00:17:08.720 no one switched. s202 00:17:08.720 --> 00:17:18.280 And it and on the flip side, we have seen teams, which is the other extreme, where they set an extremely low budget, and so that if they use a smarter model, right, they just burn it in one day. s203 00:17:18.280 --> 00:17:25.160 Yeah, then everyone just switch to the to the Dharma into the faster model Dharma and faster model because that's their budget. s204 00:17:25.160 --> 00:17:27.360 It's like almost as if the smarter model didn't exist. s205 00:17:27.360 --> 00:17:31.680 So, like be aware about your team dynamics on how they switch. s206 00:17:31.680 --> 00:17:37.000 The The best trade-off is to speed because people are naturally incentivized for it. s207 00:17:37.000 --> 00:17:41.720 So, faster, cheaper, they will just switch it to it respectively. s208 00:17:41.720 --> 00:17:43.720 Yeah. s209 00:17:44.160 --> 00:17:48.920 Once again, reminder, measure, and you measure two separate token keys. s210 00:17:48.920 --> 00:17:52.000 Uh and but this also goes beyond just measuring. s211 00:17:52.000 --> 00:17:57.720 For example, like CICD is a very common use case right now of things running in the background. s212 00:17:57.720 --> 00:18:09.400 Uh we we have helped set up for teams like faster tokens per second that that is running for the the users, like the humans, but much slower tokens per second s213 00:18:09.400 --> 00:18:13.640 um speed, but much cheaper at scale for the CICD pipeline. s214 00:18:13.640 --> 00:18:20.920 Uh and and this is just the nature of like what's your priority respectively respectively across the task. s215 00:18:20.920 --> 00:18:22.120 Yeah. s216 00:18:22.120 --> 00:18:24.960 And you can control that at a key level. s217 00:18:24.960 --> 00:18:32.320 So, another major finding is that we had experience with deploying for for teams, right? s218 00:18:32.320 --> 00:18:46.760 Uh uh So, like for example, for some companies, how many of you had the fortunate or unfortunate experience of the CEO saying, "Everyone must implement AI now, and everyone must have a second brain now." s219 00:18:46.760 --> 00:18:47.840 Yeah. s220 00:18:47.840 --> 00:18:50.280 Multiple organizations have started this. s221 00:18:50.280 --> 00:18:56.840 Uh in some cases, like even in in the UAE, it became literally mandated by by by by the law of the country. s222 00:18:56.840 --> 00:19:06.240 So, so, like a lot of departments are not prepared to know what is the trade-off respectively. s223 00:19:06.240 --> 00:19:07.920 So, a good rule of thumb, right? s224 00:19:07.920 --> 00:19:16.560 Well, developers tend to know, okay, all these smart models, what's the trade-off uh between deep seek flash and GR553, for example? s225 00:19:16.560 --> 00:19:20.760 Most uh your your finance department have absolutely no idea. s226 00:19:20.760 --> 00:19:23.640 Your your support department has absolutely no idea. s227 00:19:23.640 --> 00:19:26.960 They they'll just pick whatever's the first model and probably just go with it. s228 00:19:26.960 --> 00:19:32.040 So So, our general rule of thumb is that give department models labels, but not the model names. s229 00:19:32.040 --> 00:19:35.760 And this also gives you a lot more control as you switch it out. s230 00:19:35.760 --> 00:19:42.600 Uh what like like advised earlier, fast versus smart, but in some cases also like finance agent or Excel spreadsheet agents. s231 00:19:42.600 --> 00:19:46.760 Um these are hyper-specialized agents that were fine-tuned to those tasks respectively. s232 00:19:46.760 --> 00:19:50.560 And and this also gives you a lot more control. s233 00:19:50.560 --> 00:19:52.440 It allows you to swap up the model. s234 00:19:52.440 --> 00:20:00.680 It allows you to split split the traffic and test in between with without without telling, "Hey, users, please try this." s235 00:20:00.680 --> 00:20:05.560 The downside is also users may sometimes complain when you do that and you screw that up. s236 00:20:05.560 --> 00:20:07.840 But having that control is a good thing. s237 00:20:07.840 --> 00:20:12.240 Uh another pattern that we see, even though you do this, is maybe some of a dash preview version. s238 00:20:12.240 --> 00:20:19.000 So, that's a variant of it, but it gives you control without without taxing the communication burden across the organization. s239 00:20:19.000 --> 00:20:26.720 And this is extremely important because like like some of the teams that we work with, right? s240 00:20:26.720 --> 00:20:37.160 We literally saw members running Llama 2 models when when we are talking when talking about like in this year because no one switched the option. s241 00:20:37.160 --> 00:20:42.960 Yeah, so you really really need to need to uh need to set this in place. s242 00:20:42.960 --> 00:20:44.680 Yeah. s243 00:20:44.680 --> 00:20:45.520 All right. s244 00:20:45.520 --> 00:20:46.200 Process track. s245 00:20:46.200 --> 00:20:47.520 Process track. s246 00:20:47.520 --> 00:20:56.280 A lot of things that I mentioned earlier don't really apply as much to process track because process track tends to be a lot integrated into the applications, into the workflows, s247 00:20:56.280 --> 00:20:57.360 uh respectively. s248 00:20:57.360 --> 00:21:01.160 And and that's just by the sheer volume of the traffic. s249 00:21:01.160 --> 00:21:02.560 Um yeah. s250 00:21:02.560 --> 00:21:05.840 And And of all of teams in the process track, right? s251 00:21:05.840 --> 00:21:19.080 Uh they tend to be a lot more focused on focused on the right sizing already by default because because if you're running like trillions of tokens you most departments will not approve it like s252 00:21:19.080 --> 00:21:20.880 $10 per token million token. s253 00:21:20.880 --> 00:21:22.600 It's just not possible. s254 00:21:22.600 --> 00:21:30.600 And and so so so they start they start early with the evals, the benchmark, and then subsequently finding the model that right size fits it. s255 00:21:30.600 --> 00:21:32.800 Uh yeah. s256 00:21:32.800 --> 00:21:37.920 So So the thing is like the common question is how do I know the smaller model is good enough? s257 00:21:37.920 --> 00:21:40.640 You don't need to That's the wrong question to say. s258 00:21:40.640 --> 00:21:44.680 You just need to make sure you set up your test cases, your evals, and your examples. s259 00:21:44.680 --> 00:21:47.520 And then subsequently just do the search process from there. s260 00:21:47.520 --> 00:21:52.040 Uh companies like us will be more than happy to help you because we have over 40,000 models. s261 00:21:52.040 --> 00:21:58.520 But then But then if you're It's also not that hard to just download 10 models from Hugging Face and just try. s262 00:21:58.520 --> 00:22:07.640 Uh So So from that you you just do the search, find the cheapest model that kind of like pass most of it, and then you just see how you iterate from there, fine-tune from there. s263 00:22:07.640 --> 00:22:13.720 Uh we also have a more specific process that I don't think is unique to us that we'll cover as well also. s264 00:22:13.720 --> 00:22:18.080 So for example, like in for one of the major telcos that we work with, right? s265 00:22:18.080 --> 00:22:29.240 We actually helped their team lower their inference cost 250k to 7.6k a month switching from existing close-source model just by going through this process of like, "Okay, you're running the best model first. s266 00:22:29.240 --> 00:22:35.120 Let's see what what models we can split off, and then subsequently what's the models that we can fit in in, and then what can we fine-tune?" s267 00:22:35.120 --> 00:22:40.440 And this was This was like to uh beat the customer support use cases and and so on and so forth. s268 00:22:40.440 --> 00:22:41.360 Yeah. s269 00:22:41.360 --> 00:22:50.880 And to re-highlight back Jeff uh and simple Jeff, which is a open-source project that we did uh that replicates it. s270 00:22:50.880 --> 00:22:57.720 You'll be surprised at how many m- percentage of your AI traffic is classification traffic. s271 00:22:57.720 --> 00:23:00.120 Uh At the end of the day, right? s272 00:23:00.120 --> 00:23:01.040 Like Yeah, sure. s273 00:23:01.040 --> 00:23:03.360 Answering the customer support is text. s274 00:23:03.360 --> 00:23:04.840 But even then, right? s275 00:23:04.840 --> 00:23:05.840 Even then, right? s276 00:23:05.840 --> 00:23:07.600 A lot of teams, right? s277 00:23:07.600 --> 00:23:14.760 Already have predefined text output that they put into the context for the model to output for most cases. s278 00:23:14.760 --> 00:23:16.680 And and you could use a classifier there. s279 00:23:16.680 --> 00:23:21.040 And there's a lot This is a new new new shift within AI world. s280 00:23:21.040 --> 00:23:27.880 And that this is one of the big levers for this kind of processes in lowering your your token bill. s281 00:23:27.880 --> 00:23:29.880 Yeah. s282 00:23:30.400 --> 00:23:35.200 So, one one one example of such a model when it comes to fine-tuning. s283 00:23:35.200 --> 00:23:38.080 So, a lot of models and processes, right? s284 00:23:38.080 --> 00:23:42.520 Uh do benefit at this scale where you start to fine-tune your own model. s285 00:23:42.520 --> 00:23:50.800 Uh on the flip side, for coding models, well, there are organizations that will fine-tune models for you, like Devin, for example. s286 00:23:50.800 --> 00:23:54.160 Uh a lot of developers generally just want the best model. s287 00:23:54.160 --> 00:24:02.480 So, it's kind of unless you are in this AI space, it's kind of hard to keep pace in fine-tuning the best coding model. s288 00:24:02.480 --> 00:24:12.720 But on the flip side, for your domain, it might be worth it to fine-tune a domain model for your use case, especially if the model is much smaller. s289 00:24:12.720 --> 00:24:21.600 So, for AT&T, for example, when they fine-tuned their models, they they they went from being open class to like 27B to even 7B models. s290 00:24:21.600 --> 00:24:22.640 And they have published this. s291 00:24:22.640 --> 00:24:24.880 So, so we were one of their launch partners. s292 00:24:24.880 --> 00:24:28.960 And you can actually try their open dialogue models respectively. s293 00:24:28.960 --> 00:24:32.680 And and This is basically what they use behind the scene. s294 00:24:32.680 --> 00:24:35.280 And and it's all about volume here. s295 00:24:35.280 --> 00:24:40.120 Like, I I think I think most of you understand the scale of AT&T and the call centers and things like that. s296 00:24:40.120 --> 00:24:41.080 Uh yeah. s297 00:24:41.080 --> 00:24:46.480 But it's a good example of how you fine-tune models to to specialize use cases. s298 00:24:46.480 --> 00:24:51.760 And once you reach that point, other organizations that are major example of like Shopify have started doing this. s299 00:24:51.760 --> 00:24:57.320 And and things and and it doesn't necessarily need to be a large-scale fine-tune. s300 00:24:57.320 --> 00:25:08.240 So, for example, in the Shopify use case, they did it because Shopify web store had a proprietary la- programming language for how to configure the the web store. s301 00:25:08.240 --> 00:25:12.240 And the existing models just weren't that great at it, so they went to fine-tune it. s302 00:25:12.240 --> 00:25:19.840 Uh but and AT&T also they had examples of that uh in particular on like how to arrange the the cell towers and things like that. s303 00:25:19.840 --> 00:25:20.320 Yeah. s304 00:25:20.320 --> 00:25:22.120 Uh very telco-specific task. s305 00:25:22.120 --> 00:25:24.600 I might not be the right expert to answer that question. s306 00:25:24.600 --> 00:25:26.760 Uh but they had those use cases. s307 00:25:26.760 --> 00:25:28.840 So so they fine-tuned respectively. s308 00:25:28.840 --> 00:25:31.760 You too may have that those use cases. s309 00:25:31.760 --> 00:25:34.280 And how do you bootstrap such a model, right? s310 00:25:34.280 --> 00:25:46.120 Is that the good news is that uh you can easily bootstrap things these days because if let's say you're willing uh if the best model that is too expensive to run at scale s311 00:25:46.120 --> 00:25:49.640 can do the task, you can already get started. s312 00:25:49.640 --> 00:25:52.920 So so so an easy way to do this, right? s313 00:25:52.920 --> 00:25:58.760 Uh that we see very commonly is like either you generate your own data or prepare your own data, which is hard. s314 00:25:58.760 --> 00:26:01.080 No one likes to do that for some reason. s315 00:26:01.080 --> 00:26:11.880 But that but another common path is that they start with an expensive model and then they deploy in a small scale and then they generate they they generate data set. s316 00:26:11.880 --> 00:26:20.200 I understand that some labs it's against their terms of service to do that, so you may want to check with your lawyers, but there are labs that openly embrace it. s317 00:26:20.200 --> 00:26:26.680 So, for example, Google Gemini uh their their frontier models, they literally have a dedicated API service for this. s318 00:26:26.680 --> 00:26:37.840 Uh so so so this allows you to build data sets respectively on your use case on a strong model, which you can then which which you can then reduce s319 00:26:37.840 --> 00:26:40.160 dramatically to let's say a 7B scale. s320 00:26:40.160 --> 00:26:49.880 Uh alternatively, open-source models are more than happy for you to to to run and distill on, so you can run the best uh open-source models respectively as well. s321 00:26:49.880 --> 00:26:57.120 Once you collect them, you can then fine-tune the data and then you can fine-tune and respectively like different sizes to see what fits best. s322 00:26:57.120 --> 00:27:03.240 So, once again, like in the 80/20 example, they have the 27B and 7B for different levels of the call center. s323 00:27:03.240 --> 00:27:07.240 Like your routine your routine calls may come in at a Wait. s324 00:27:07.240 --> 00:27:16.800 I don't think I can say which size goes where, but your routine calls can come in for the much smaller model, and then it escalates up due to some classifier triggers as an example. s325 00:27:16.800 --> 00:27:24.400 Then then so so so that's how you get the traffic to flow, get the traffic split, and then you get it at that efficient level. s326 00:27:24.400 --> 00:27:38.520 Uh one of the things that we also tend to work with teams on on when doing this is that even even when they go through the stages and they completely split over their traffic to the fine-tuned model, because the best of the best models still progress, s327 00:27:38.520 --> 00:27:47.600 uh they get they uh we typically advise them to cut over a cut a part of their traffic as a shadow traffic to the best model to keep building the data set. s328 00:27:47.600 --> 00:27:54.840 And and and this allows you to ride the wave and progress of the best open models respectively. s329 00:27:54.840 --> 00:28:06.000 Uh and the unit economics is actually very important here, because like one thing that I would like to remind people, right, is people severely underestimate the the the speed s330 00:28:06.000 --> 00:28:11.120 that AI is going, especially for models that you can run much cheaper at scale. s331 00:28:11.120 --> 00:28:13.600 Like you take this laptop as an example. s332 00:28:13.600 --> 00:28:20.600 I like to measure, right, what is the best model that I can buy in the most expensive MacBook Pro here. s333 00:28:20.600 --> 00:28:30.960 And today's best models, be it MoE models at 31B, 32B, 27B size, is already much better than GPT-4. s334 00:28:30.960 --> 00:28:38.800 So, essentially, this model that can run on a laptop is already better than the best model that you have last year. s335 00:28:38.800 --> 00:28:39.360 Give and take. s336 00:28:39.360 --> 00:28:42.440 People like to argue on the specifics, but let's just say broadly. s337 00:28:42.440 --> 00:28:55.480 And when you fine-tune it to your task, that means if you can run that model on a laptop respectively, you can run it on on on your GPU servers at scale much more efficiently. s338 00:28:56.040 --> 00:28:57.240 Yeah. s339 00:28:57.240 --> 00:28:58.800 So, Yeah. s340 00:28:58.800 --> 00:29:05.120 So, that brings in to the next topic, buying GPUs which which which is apparently increasingly a challenge. s341 00:29:05.120 --> 00:29:08.480 So, yeah, that's where companies like us come in. s342 00:29:08.480 --> 00:29:18.520 One of the things that we we realized through a lot of customer conversations, people are rather surprised that things don't scale linearly. s343 00:29:18.520 --> 00:29:28.960 Uh what I mean by that is like there is a huge trade-off between tokens per second on an individual request to to the cost to the cost of the request itself. s344 00:29:28.960 --> 00:29:44.080 So, for example, if you do not care about about the cost, you could run the best the model, let's say a fixed model, on the best hardware, on all eight GPUs, batch size one, and you could get, let's say, 100 tokens per second, 200 tokens per second. s345 00:29:44.080 --> 00:29:54.000 If you like if you really want and and and this is not our specialization, like you really want like thousands of tokens per second, that's where companies like Cerebras is more than happy to offer that as well. s346 00:29:54.000 --> 00:29:55.480 And it costs even more. s347 00:29:55.480 --> 00:29:59.080 So, there are definitely use cases where this makes sense. s348 00:29:59.080 --> 00:30:05.360 But, it's but a lot of people fall into the trap of thinking, "Hey, so if I have the speed, maybe it's half the cost." s349 00:30:05.360 --> 00:30:07.000 It's not even close. s350 00:30:07.000 --> 00:30:13.960 Like like a rule of thumb that I sometimes say like, "For every two x of speed, it's three x the cost and it compounds." s351 00:30:13.960 --> 00:30:23.560 In this specific example for JM 5.3, it's like one JM node is like you can see the same server, 112 tokens per second for running individual requests. s352 00:30:23.560 --> 00:30:32.000 But, if you run 16 requests in parallel, you get about half the speed for all the requests, but you literally 10x the output total output. s353 00:30:32.000 --> 00:30:37.680 You And and that that gives you like literally around 9 to 10x lower in cost. s354 00:30:37.680 --> 00:30:41.480 In just like changing your changing your target speed profile. s355 00:30:41.480 --> 00:30:45.080 And so, when a lot of customers ask us like, "How do we deploy things at scale?" s356 00:30:45.080 --> 00:30:49.000 Sometimes the first question is, "What's the model quality you want? s357 00:30:49.000 --> 00:30:51.200 And what's the tokens per second speed that you want?" s358 00:30:51.200 --> 00:30:54.760 And then we start recommending the amount GPUs, the amount deployment. s359 00:30:54.760 --> 00:31:00.240 And this this target tokens per second is an important trade-off. s360 00:31:00.240 --> 00:31:03.480 Essentially, um in more extreme cases, right? s361 00:31:03.480 --> 00:31:10.360 If you really, really can run things at 10 tokens per second, rarely anyone does, but there are some weird use cases that allows it. s362 00:31:10.360 --> 00:31:15.000 You can get even more incredible to- total throughput volumes, respectively. s363 00:31:15.000 --> 00:31:25.600 Typically, we see customers choosing between 20 30, which is kind of like uh your chat GPT speed without without too much thinking, all the way until like 60, which is like the fastest speed. s364 00:31:25.600 --> 00:31:29.360 And if they want 120 tokens like it tends to be this this range. s365 00:31:29.360 --> 00:31:35.000 Most stick between 20 30 to 60, depending on how how long the chain of task is. s366 00:31:35.000 --> 00:31:37.000 Yeah. s367 00:31:37.000 --> 00:31:46.400 So, yeah, uh I kind of mentioned you generally only want to pay for the tokens per second speed uh when when it's in front of someone. s368 00:31:46.400 --> 00:31:50.880 Uh whe- whether it's in front of your developers or whether it's in front of your customers. s369 00:31:50.880 --> 00:31:52.600 Those are the trade-offs that you want to make. s370 00:31:52.600 --> 00:31:53.880 And even within then, right? s371 00:31:53.880 --> 00:31:58.000 We have seen cases where even just a slight tweak down, right? s372 00:31:58.000 --> 00:32:05.040 Can easily save two times the cost, but has near minimal impact for the cus- uh customer, respectively. s373 00:32:05.040 --> 00:32:07.040 Yeah. s374 00:32:07.160 --> 00:32:10.280 And once again, background jobs, please don't run it at 120 tokens per second. s375 00:32:10.280 --> 00:32:13.240 It makes no sense when you do that. s376 00:32:13.320 --> 00:32:14.560 Yeah. s377 00:32:14.560 --> 00:32:18.400 So, uh once you have Previously, I said measure, right? s378 00:32:18.400 --> 00:32:22.320 You're able to see your your your traffic pattern. s379 00:32:22.320 --> 00:32:30.480 So, a lot of traffic patterns tend to be bursty in some nature with some base load because someone's always running something at some time, uh especially background agents. s380 00:32:30.480 --> 00:32:36.240 And and this is where you can you can also plan your deployments, respectively. s381 00:32:36.240 --> 00:32:48.520 Uh I know it since we're are doing financial a lot of financial may not have this as an option, but because they are only dedicated instance deployment, but if for some cases where people allow mixed where they have dedicated instances s382 00:32:48.520 --> 00:32:53.240 with some burst to the public cloud, this is where you have some flexibility respectively. s383 00:32:53.240 --> 00:32:59.840 So, like the biggest cost savings you have right now in the market is deploying a dedicated instance if you know your base workload. s384 00:32:59.840 --> 00:33:03.560 So, as long as you fully use it, you kind of have a savings there. s385 00:33:03.560 --> 00:33:10.280 But, it's a bit more complicated depending on your traffic pattern, and a lot of these things can be tested and validated uh respectively. s386 00:33:10.280 --> 00:33:13.160 And this is like essentially what any inference company will work with you. s387 00:33:13.160 --> 00:33:14.320 It's not unique to us. s388 00:33:14.320 --> 00:33:16.680 We'll be like, "Hey, this is Do you know your traffic workload? s389 00:33:16.680 --> 00:33:18.040 Can you share some data?" s390 00:33:18.040 --> 00:33:20.600 And then subsequently, we'll be like, "Okay, we propose this much. s391 00:33:20.600 --> 00:33:21.920 You can try switching to this." s392 00:33:21.920 --> 00:33:26.360 And then we we generate we can generally show you your savings respectively. s393 00:33:26.360 --> 00:33:38.280 And in some extreme cases where you have a lot of spikes, you may want to have limit your dedicated capacity to a certain amount and let the spikes spill over to the public cloud respectively. s394 00:33:38.280 --> 00:33:49.600 The different use cases uh have different right sizing, but this is something that by measuring your token workload once again at the beginning, you can start exploring these options, s395 00:33:49.600 --> 00:33:51.880 and the savings can be huge. s396 00:33:51.880 --> 00:34:02.200 Like once again, like for the customer that we served two 2 trillion token, and that was like uh 4.8 Opus class model, they are paying less than 0.0 $3 s397 00:34:02.200 --> 00:34:03.160 per million token. s398 00:34:03.160 --> 00:34:06.280 Like you can't get that price anywhere. s399 00:34:06.280 --> 00:34:08.040 Yeah. s400 00:34:08.040 --> 00:34:09.360 So, yeah. s401 00:34:09.360 --> 00:34:12.280 Simple disclaimer is the spikes really matter. s402 00:34:12.280 --> 00:34:22.800 Uh it's very rare that someone has a short spike at the extreme end where they like run a workload every month at that scale, and then nothing else matter. s403 00:34:22.800 --> 00:34:26.280 So, so usually some base workload exists. s404 00:34:26.280 --> 00:34:28.240 It's just how big of a base workload. s405 00:34:28.240 --> 00:34:30.240 Yeah. s406 00:34:31.200 --> 00:34:32.159 So, yeah. s407 00:34:32.159 --> 00:34:34.800 Then the last one is check your processes. s408 00:34:34.800 --> 00:34:38.760 Uh this has nothing to do with inference optimization from here onwards. s409 00:34:38.760 --> 00:34:45.159 Um Some slides may sound that very critical of AI, but I'm generally an optimist. s410 00:34:45.159 --> 00:34:51.040 Uh just this I just want to lay that out because things can be quoted out of context. s411 00:34:51.040 --> 00:35:06.800 So, one of the uh Pandora's Box question or problematic question that we are starting to see is that a lot of organizations starting to ask, "Hey, I'm s- 10x 100x in my token spend, but I'm not seeing results s412 00:35:06.800 --> 00:35:08.360 at that same scale." s413 00:35:08.360 --> 00:35:23.480 Um respect- like like we are starting to see reports from from various organizations like uh AI where AI has limited productivity only has 6% productivity or 7%, which is not the scale that a lot of us were promised or we envision. s414 00:35:23.480 --> 00:35:33.640 And and unfortunately, this is the case because we are at the stage where everyone's just taking a giant hammer and hammering everything. s415 00:35:33.640 --> 00:35:39.520 Um it's not that AI is not useful, it's just that we tend to just spam it everywhere. s416 00:35:39.520 --> 00:35:46.200 And yeah, if uh there's a s- there are times where I just want to like, "Can you please not do this as another co-pilot?" s417 00:35:46.200 --> 00:35:48.520 Because it's just too much of it. s418 00:35:48.520 --> 00:35:52.160 You need to think of how you use it respectively. s419 00:35:52.160 --> 00:35:54.600 Uh um yeah. s420 00:35:54.600 --> 00:36:06.880 And and another major issue that we start to realize is that well well AI's are are being deployed at a larger scale and more frequently, and people get faster in creating stuff, s421 00:36:06.880 --> 00:36:11.840 the organization processes may not go as fast. s422 00:36:11.840 --> 00:36:18.920 The first bottleneck for for coding coding environments will be the CICDs, and you all may heard that CPU becomes a bottleneck respectively. s423 00:36:18.920 --> 00:36:24.560 Uh another another thing is like is just the processes in place uh respectively. s424 00:36:24.560 --> 00:36:25.400 How do you trust? s425 00:36:25.400 --> 00:36:26.520 How do you review? s426 00:36:26.520 --> 00:36:29.200 I think the previous speaker spoke at length at trust. s427 00:36:29.200 --> 00:36:32.320 Yeah, the lack of trust is what slows things down. s428 00:36:32.320 --> 00:36:34.400 And this could be rightfully so. s429 00:36:34.400 --> 00:36:38.520 Maybe the agent the the AI agent was badly designed and shouldn't be trusted. s430 00:36:38.520 --> 00:36:39.960 But it also flip and inverse. s431 00:36:39.960 --> 00:36:44.040 What if hypothetically the that agent was doing everything perfectly? s432 00:36:44.040 --> 00:36:50.200 But people weren't trusting it because it didn't have the test in place, it didn't have the validation, and it didn't build that trust. s433 00:36:50.200 --> 00:36:51.640 That slows things down. s434 00:36:51.640 --> 00:36:57.560 And these things make and these things have a much bigger impact. s435 00:36:58.960 --> 00:37:07.160 So, once again, be very wary about how the hammer is being used because you need to make your adjustments accordingly. s436 00:37:07.160 --> 00:37:12.320 So, and this goes back into saying like tokens are not work done. s437 00:37:12.320 --> 00:37:28.200 You have uh um um and one major example that I will highlight, right, is is this happens more often than than I would like, is that sometimes spamming more agents and more tokens sometimes means you get the same task done multiple times. s438 00:37:28.200 --> 00:37:47.520 So, one of the anti-patterns that we started seeing, right, uh we are well in overall the engineering organization improve productivity by clearing more issues quickly, automating clearing of issues, for example, like when someone like when a QA and and things like QA teams reporting an issue and the agent automatically fixing it without even an engineer, s439 00:37:47.520 --> 00:37:48.680 those are great. s440 00:37:48.680 --> 00:37:56.000 But one of the one of the common issues that we start facing is that people get ambitious really fast. s441 00:37:56.000 --> 00:38:03.840 And they instead of doing small changes, they start doing huge changes like changing like half the code bases or one quarter of the code bases. s442 00:38:03.840 --> 00:38:06.720 Sometimes when this is planned with proper testing, it's great. s443 00:38:06.720 --> 00:38:15.560 But a lot of times, right, the change is happening at such a large scale, right, the bottleneck becomes the on the reviewer to merge the code. s444 00:38:15.560 --> 00:38:28.240 And because because everything is done on such a large scale where a human cannot process it, one of the common anti-patterns that we start seeing is that okay, someone has implemented the the there's issues being done. s445 00:38:28.240 --> 00:38:36.560 And because there's so much code being generated, issues that do not have code generated do not get reviewed and do not get picked up by the reviewer. s446 00:38:36.560 --> 00:38:39.120 So, someone needs to generate the code using an agent. s447 00:38:39.120 --> 00:38:50.640 And then when the agent generate that giant pile of code, which should be a lot less if they were to like properly review prompting and stuff, the reviewer look at it and it's like, "Huh, this is too many code. s448 00:38:50.640 --> 00:38:55.560 Can I ask my agent to highlight what changes are relevant and then re-implement it?" s449 00:38:55.560 --> 00:39:01.080 And and and in the end they started rebuilding it by hand with the with the agent. s450 00:39:01.080 --> 00:39:04.160 So, they literally discard the first agent's job. s451 00:39:04.160 --> 00:39:05.960 There's multiple good reasons why this happened. s452 00:39:05.960 --> 00:39:19.520 For example, like reviewers want to maintain control and quality of the main code base, but this is a very common example of a bottleneck where you where where if you didn't think the process through, you're just throwing more and more things. s453 00:39:19.520 --> 00:39:22.960 And respectively, and the speed is not increasing. s454 00:39:22.960 --> 00:39:25.880 So, review your processes. s455 00:39:25.880 --> 00:39:33.440 A good way around this is naturally to to to instill better testing practices and so that you can automate that and filter out. s456 00:39:33.440 --> 00:39:42.080 Also, reminding teams to like, "Hey, split up your your your merge or or make it more modular to make it easier for the review process." s457 00:39:42.080 --> 00:39:42.760 Like things like that. s458 00:39:42.760 --> 00:39:44.280 These are human processes. s459 00:39:44.280 --> 00:39:45.760 It has nothing to do with the AI. s460 00:39:45.760 --> 00:39:48.320 And yeah, we tend to get too excited about AI sometimes. s461 00:39:48.320 --> 00:39:53.480 So, this is not just a code problem. s462 00:39:53.480 --> 00:40:03.720 Uh we are starting to see like because uh AI agents are being deployed to make decks, this deck was entirely design-wise was generated by AI. s463 00:40:03.720 --> 00:40:05.920 I went through multiple iterations. s464 00:40:05.920 --> 00:40:07.200 It's not a zero-shot. s465 00:40:07.200 --> 00:40:14.920 I I went through every page like at least 30 times to to change the text, but the design was done by agents. s466 00:40:14.920 --> 00:40:22.240 And and sometimes within the organization you're starting to notice people that just one shot from the entire slides, things like that. s467 00:40:22.240 --> 00:40:24.360 And and they start passing it around. s468 00:40:24.360 --> 00:40:25.560 They start doing meetings. s469 00:40:25.560 --> 00:40:30.640 And then it's on everyone else to filter what's the inconsistency, what's uh what's the misalignment? s470 00:40:30.640 --> 00:40:34.040 Why is this two data points completely different? s471 00:40:34.040 --> 00:40:45.560 This is also another example of like same as the previous example, people generating code or people generating slides or people generating report and the burden shifts onto a some human down the road to verify things. s472 00:40:45.560 --> 00:40:48.880 So, be very aware about this organization change. s473 00:40:48.880 --> 00:40:58.240 Uh why am I still an optimist is that I I don't think this is necessarily a bad thing in itself because the problem is everyone needs to learn. s474 00:40:58.240 --> 00:41:02.240 And this is where we just start adapting our processes accordingly. s475 00:41:02.240 --> 00:41:08.600 Uh and the other major thing is that well, there are a lot of common examples on how to use AI. s476 00:41:08.600 --> 00:41:12.600 You'll know your subject domain for your organization better. s477 00:41:12.600 --> 00:41:24.320 And there will be use cases where an AI could be used in place to automate a process, for example, that could save your organization uh millions of dollars or increase your productivity at a grand scale s478 00:41:24.320 --> 00:41:30.360 that that only you will know and not me as an AI expert will come in. s479 00:41:30.360 --> 00:41:36.560 I like to point to the most uh famous example that Walmart had where was it Walmart? s480 00:41:36.560 --> 00:41:37.400 I believe it's Walmart. s481 00:41:37.400 --> 00:41:42.920 Where they literally started putting a stool a stepping stool inside every single truck. s482 00:41:42.920 --> 00:41:51.800 And that was like the simplest change, but it lowered their bill by the millions because it lowered back injuries across for heavy goods respectively. s483 00:41:51.800 --> 00:41:56.320 It it's some something specific like that with the use of AI. s484 00:41:56.320 --> 00:42:01.920 And and it may not even necessarily be a the best AI model, but actually can have the biggest impact. s485 00:42:01.920 --> 00:42:03.320 Yeah. s486 00:42:03.320 --> 00:42:03.720 Yeah. s487 00:42:03.720 --> 00:42:12.160 So, uh doubling down on my previous point, uh one of the term that that was coined by uh that that's been trending is called the slot grenade. s488 00:42:12.160 --> 00:42:16.640 So, generating lots of things in AI and then leaving it to someone else. s489 00:42:16.640 --> 00:42:18.920 Do not let slop grenades happen. s490 00:42:18.920 --> 00:42:22.960 Uh and this is a culture thing and a processing. s491 00:42:22.960 --> 00:42:24.200 Yep. s492 00:42:24.200 --> 00:42:29.560 So, the some guideline rules that I would set and this is more subjective. s493 00:42:29.560 --> 00:42:31.280 People people disagree. s494 00:42:31.280 --> 00:42:37.680 Um to prevent a slop grenade from happening, enforce that people should own what they hand over or generate. s495 00:42:37.680 --> 00:42:42.480 So, for example, this slide, I own every text that is generated. s496 00:42:42.480 --> 00:42:44.520 If there's an issue, that's on me. s497 00:42:44.520 --> 00:42:48.160 It's not my AI agent who who who has an issue. s498 00:42:48.160 --> 00:42:50.240 I do not get to hand it off to an agent. s499 00:42:50.240 --> 00:42:51.680 I must own the responsibility. s500 00:42:51.680 --> 00:42:54.840 Same thing for code, same thing for anything that is generated. s501 00:42:54.840 --> 00:42:58.480 And you also have to measure outcomes, not usage. s502 00:42:58.480 --> 00:43:15.440 Uh And so this gets more complicated because I I don't want to explain too too long to this, but like there there is like this simple formula that's that I think people take as a rule of thumb, tokens times price divided by outcomes someone actually use. s503 00:43:15.440 --> 00:43:18.400 And and and this help guide a lot of things. s504 00:43:18.400 --> 00:43:21.920 Smaller, more efficient models, faster model, bigger model, yada yada. s505 00:43:21.920 --> 00:43:34.080 But, what I don't like about this is that another analogy that I'll use about AI is think of it more as like supplements or steroids that enhance the team. s506 00:43:34.080 --> 00:43:38.200 Think of it as uh like if let's say I use construction work as an example. s507 00:43:38.200 --> 00:43:41.920 It let's say I can use it to increase the muscle mass for the workers. s508 00:43:41.920 --> 00:43:44.160 For example, not that it's ethically to do so. s509 00:43:44.160 --> 00:43:47.640 I know that's not the case, but let's just say as an example in the train of thought. s510 00:43:47.640 --> 00:43:51.720 Some teams will have it and they improve productivity, some teams don't. s511 00:43:51.720 --> 00:43:57.920 But, even without that, it's quite clear some teams are much better than others. s512 00:43:57.920 --> 00:44:04.240 And and so the AI being a supplement here, right, doesn't necessarily change everything. s513 00:44:04.240 --> 00:44:16.320 So, you need to be aware that outcomes outcomes are hard to measure respectively, but it's something that you can start measuring and observing most importantly back to the first step. s514 00:44:16.320 --> 00:44:17.560 Measure. s515 00:44:17.560 --> 00:44:23.400 You know the tokens spent per department, per team, and you can start seeing which team is performing better and you start seeing how much tokens they use. s516 00:44:23.400 --> 00:44:26.280 You can start seeing what works. s517 00:44:26.280 --> 00:44:27.240 Yeah. s518 00:44:27.240 --> 00:44:28.720 Yeah, as I said, this is flawed measurement. s519 00:44:28.720 --> 00:44:29.720 So, yeah. s520 00:44:29.720 --> 00:44:33.720 Uh that's that's the high-level overview respectively. s521 00:44:33.720 --> 00:44:43.640 And if if you want to to like kickstart your process internally within your organization, you can you can either talk to me or my team or like you have six questions for you to start asking like s522 00:44:43.640 --> 00:44:54.840 who team spent what, what's the what what must you test to use a cheaper model, what's your base load, what's the default model for each team, and which workload has a person waiting that you need something faster, yeah. s523 00:44:54.840 --> 00:44:57.480 And how much of what we generate is useful. s524 00:44:57.480 --> 00:45:02.640 These are like guideline questions for you to start asking to refine your tokenomics. s525 00:45:02.640 --> 00:45:03.200 Yeah. s526 00:45:03.200 --> 00:45:05.760 So, Q&A. s527 00:45:07.080 --> 00:45:07.920 Yes. s528 00:45:07.920 --> 00:45:11.200 Yeah, so when you mentioned fine-tuning models, is that s529 00:45:11.200 --> 00:45:12.360 Test, test. s530 00:45:12.360 --> 00:45:13.200 Um s531 00:45:13.200 --> 00:45:13.800 Yeah, I'm trying. s532 00:45:13.800 --> 00:45:22.880 like SFT sort of fine-tuning or is there like other sort of uh fine-tuning methods that you have depending on what specific use case there is or what data there is? s533 00:45:22.880 --> 00:45:36.080 Uh okay, so in general, it tends to be SFT, but I don't think um Well, we do provide fine-tuning services and it tends to be SFT. s534 00:45:36.080 --> 00:45:40.840 I there is a lot of methods that are happening right now currently. s535 00:45:40.840 --> 00:45:50.440 I know RL is very sexy right now, especially in the coding environment, because it's easy to generate the data in the loop, but what we realize for a lot of organizations, s536 00:45:50.440 --> 00:46:00.360 especially like in New York, for example, financial institute, things like that, there's not really an RL environment for financial task, for example. s537 00:46:00.360 --> 00:46:04.800 And and so it tends to be more SFT base. s538 00:46:05.600 --> 00:46:20.720 And and there are also other organizations like adapted adoption AI where where they are starting to also automate the the the refinement of the data sets for you to train the model be SFT base or thinking s539 00:46:20.720 --> 00:46:23.360 thinking of machine base. s540 00:46:23.360 --> 00:46:26.000 Focusing on this might be the wrong question. s541 00:46:26.000 --> 00:46:32.120 It to me generally what people should focus on is the evals respectively. s542 00:46:32.120 --> 00:46:43.880 So here's a good example for Jeff that came when the Jeff model came out and then we did the open source project called simple Jeff which replicates the Jeff API for existing open source model. s543 00:46:43.880 --> 00:46:48.640 A lot of people quickly realized that the Quen 27B was really good at this task. s544 00:46:48.640 --> 00:46:52.720 And and there were multiple teams that went to fine tune the Quen 27B. s545 00:46:52.720 --> 00:47:00.240 I on on our side because we are trying to take the approach like our company like we want to accept any model and then we just build the API on top of it. s546 00:47:00.240 --> 00:47:05.720 We started taking the approach of like hey we see the benchmark we see the existing model why don't we prompt tune it instead. s547 00:47:05.720 --> 00:47:07.320 It's not even fine tune. s548 00:47:07.320 --> 00:47:14.280 And then and then two weeks later as hugging face went to do all the benchmarks at scale like over 10,000 questions things like that. s549 00:47:14.280 --> 00:47:20.160 Somehow my prompt tuned Quen 27B beat most of the 27B fine tune. s550 00:47:20.160 --> 00:47:30.000 So this this is not a flex because I'm quite sure now my model is beat by the recent Cloudflare fine fine tune model that was done on a more proper basis s551 00:47:30.000 --> 00:47:40.560 but it's more of like the tuning method might the tuning method might not be the right question it's more like what's your evals and what's the results that lead to it. s552 00:47:40.560 --> 00:47:50.800 And follow for that do you need do you need to collect golden traces for the set of data you need if you're doing SFT or whatever it is or can you do s553 00:47:50.800 --> 00:47:55.760 it let's just say like your your prior task the frontier models get like 80% accurate right? s554 00:47:55.760 --> 00:47:58.440 So maybe 20% of the time it makes some sort of mistake. s555 00:47:58.440 --> 00:48:03.120 If you fine-tune on that, won't the fine-tuned model also get like 20% of it wrong? s556 00:48:03.120 --> 00:48:11.400 Are you supposed to filter that out somehow and get only the golden traces for training, or can you just throw all the traces in and train? s557 00:48:11.400 --> 00:48:20.880 So, this is more of a personal opinion technique because uh so, the generally, I try to split the data set because people are lazy filter. s558 00:48:20.880 --> 00:48:24.880 Um so, I try to at least get a golden trace data set. s559 00:48:24.880 --> 00:48:35.600 And the golden trace data set maybe is much smaller uh respectively, and we just and we then we fine-tune that with a much heavier weightage uh as well. s560 00:48:35.600 --> 00:48:50.840 But, there have been multiple papers especially as especially with thinking tokens that even the non- uh the failed task, right? s561 00:48:50.840 --> 00:48:59.800 The as much smaller model, when you fine-tune, tends to benefit from the training because it improved the thinking process even if it got the wrong answer. s562 00:48:59.800 --> 00:49:06.480 Um this is something This is really a good case of like try and then measure, and it works 70% of the time. s563 00:49:06.480 --> 00:49:11.240 Sometimes, you just you need to increase the golden trace ratio, but it's a general rule of thumb. s564 00:49:11.240 --> 00:49:21.760 I think for a lot of customers when they're trying to to SFT down a smaller model, the general rule of thumb they're trying to keep is can I keep to the near same level of performance, s565 00:49:21.760 --> 00:49:28.560 and maybe the eval is down 1% or even in some cases it improved by 10% because the base model wasn't specialized for these tasks. s566 00:49:28.560 --> 00:49:33.160 And more importantly, does it 100x reduce my bill? s567 00:49:33.160 --> 00:49:37.240 So, that's usually the motivation respectively. s568 00:49:37.240 --> 00:49:42.240 Yeah, golden samples are critical if you want to do beyond the baseline. s569 00:49:42.240 --> 00:49:52.360 And sometimes that requires humans to actually look at the data, which as much as I love my customers to look at the data more, people don't like to do that. s570 00:49:55.800 --> 00:49:56.520 Hi. s571 00:49:56.520 --> 00:50:03.360 Um how would you deal with uh code spend that would mostly come from Claude's code and all the integrations there? s572 00:50:03.360 --> 00:50:09.000 Cuz then your model is fully locked into the Claude uh ecosystem, I suppose. s573 00:50:09.000 --> 00:50:14.080 Do you have any advice on how to deal deal with those type of situations? s574 00:50:14.080 --> 00:50:17.619 Uh yeah, unfortunately Claude code is the example of fully locked in. s575 00:50:17.619 --> 00:50:17.960 [laughter] s576 00:50:17.960 --> 00:50:26.120 So, uh sometimes sometimes the first movement is actually getting um them able to switch the harness. s577 00:50:26.120 --> 00:50:30.080 Um harness it seems to be a lot more flexible thing right now. s578 00:50:30.080 --> 00:50:33.320 Like we have seen organizations that are like fully locked in. s579 00:50:33.320 --> 00:50:36.200 Like I'm only going to use Devin only or Mboni. s580 00:50:36.200 --> 00:50:40.560 We have seen organizations that um that are more flexible. s581 00:50:40.560 --> 00:50:44.120 Like literally they are running their they are creating their own agent harness. s582 00:50:44.120 --> 00:50:46.200 And I'm my response is like, "Why? s583 00:50:46.200 --> 00:50:48.280 Why are you creating your own agent harness from scratch?" s584 00:50:48.280 --> 00:50:52.680 Uh but like a common one that we see people switching to is for example Pi. s585 00:50:52.680 --> 00:50:54.840 And this is also because there's a lot of excitement around it. s586 00:50:54.840 --> 00:51:02.520 People have shown that even with the same model, it gets a much cheaper cost per token because of the way it's simple and efficient. s587 00:51:02.520 --> 00:51:10.800 Um and from there on, the I think the beauty for the Pi coding agent harness is that it's easy to adapt because it's so si- so lightweight. s588 00:51:10.800 --> 00:51:15.120 We've seen a lot of teams that just start prompting their agents to make modifications. s589 00:51:15.120 --> 00:51:21.600 And once once they have that, then then they start getting the cycle of experimenting and changing. s590 00:51:21.600 --> 00:51:33.560 Uh a common way to get them to be very incentivized to change is to actually introduce a a a good enough model that is really fast, like 100 tokens per second. s591 00:51:33.560 --> 00:51:35.760 And then suddenly they start giving that a try. s592 00:51:35.760 --> 00:51:40.400 And some people start or like some of the first shift would be they still use their Claude code as their planner. s593 00:51:40.400 --> 00:51:45.280 And then they instruct the Claude code, "Use this Pi agent with this model to do the implementation." s594 00:51:45.280 --> 00:51:48.040 Hence the planner implementation split. s595 00:51:48.040 --> 00:51:51.680 It's more yeah, and that's where the transition started to happen. s596 00:51:51.680 --> 00:51:55.960 It's not easy to change people's agents, but yeah. s597 00:51:55.960 --> 00:52:07.600 Um Yeah, so an example of these could be perhaps to expose this this smaller agent as an MCP in inside cloud or having these tools where you can offload s598 00:52:07.600 --> 00:52:12.800 I I suppose is that something that you would you you would say could work? s599 00:52:12.800 --> 00:52:19.680 MCP is more tricky because MCP agents unless done right don't have access to your file system. s600 00:52:19.680 --> 00:52:25.040 So typically it's just something way simpler like pies installed and they run the batch command pie this. s601 00:52:25.040 --> 00:52:27.240 Yeah, that that's usually the case. s602 00:52:27.240 --> 00:52:28.920 Thanks. s603 00:52:28.920 --> 00:52:30.920 Yeah. s604 00:52:36.640 --> 00:52:51.160 Can you share if possible any case studies about caching solutions being very thoughtful about caching and the outcomes that have been generated for specific customers even if it's anonymized? s605 00:52:52.720 --> 00:52:54.240 What's the specific challenge you're talking about? s606 00:52:54.240 --> 00:53:03.640 You do one caching you Okay, is your question more like you want anonymized you do Wait, you do not want customer private information to save in the cache. s607 00:53:03.640 --> 00:53:15.040 No, so so like with customers that you've worked with have you got any case studies you can share about the outcomes that they've been able to generate because of being thoughtful s608 00:53:15.040 --> 00:53:20.280 about how they cache and different sort of caching methodologies? s609 00:53:20.640 --> 00:53:27.240 Uh I think this one we can take offline, but in general for our larger scale customers we deploy them with dedicated cache. s610 00:53:27.240 --> 00:53:33.360 So it kind of solves a lot of PI issues because it's the the cache is only for that customer. s611 00:53:33.360 --> 00:53:40.400 Um yeah, and obviously there's prompt formatting, but I may need to dig in more specifically like what you're trying to ask here. s612 00:53:40.400 --> 00:53:44.040 The short answer is yes, but there's a lot of way to interpret this. s613 00:53:44.040 --> 00:53:44.320 Yeah. s614 00:53:44.320 --> 00:53:45.440 Cool. s615 00:53:45.440 --> 00:53:47.800 Sorry about that. s616 00:53:49.080 --> 00:53:54.520 Uh thoughts on turn or task-based auto routing? s617 00:53:55.000 --> 00:54:00.240 Oh, um I'm a I'm generally more towards task-based. s618 00:54:00.240 --> 00:54:06.920 Um mostly because it's much easier to control expectation at a department organization level. s619 00:54:06.920 --> 00:54:14.760 So, this this is back to like, hey, a department may have a label model and I kind of auto router that model for them. s620 00:54:14.760 --> 00:54:19.760 Uh I know there's a lot of hype right now for turn-based auto routing. s621 00:54:19.760 --> 00:54:27.400 One of the issues with turn-based auto routing is that you need to lock down your your turn-based auto routing to let's say at most two to three models. s622 00:54:27.400 --> 00:54:30.800 If not, your cache will be extremely inefficient to be used. s623 00:54:30.800 --> 00:54:35.760 So, um so for example, we have seen turn-based for GRM 5.3 and GRM 5.3 flash. s624 00:54:35.760 --> 00:54:36.640 Make sense. s625 00:54:36.640 --> 00:54:46.640 But if you put in 30 models and it keeps changing your model every turn, your your low cache hit rate will start to wreck your your optimization. s626 00:54:46.640 --> 00:54:47.920 Yeah, that's what I've seen before. s627 00:54:47.920 --> 00:54:51.480 I saw I saw it like request level and it just didn't make sense and then s628 00:54:51.480 --> 00:54:51.840 Yeah. s629 00:54:51.840 --> 00:54:54.120 But two models tends to be okay. s630 00:54:54.120 --> 00:54:57.400 Uh task and organization makes the most sense. s631 00:54:57.400 --> 00:54:57.720 Yeah. s632 00:54:57.720 --> 00:54:58.000 Yeah. s633 00:54:58.000 --> 00:55:00.160 Yeah, I agree. s634 00:55:01.520 --> 00:55:03.600 All right. s635 00:55:05.360 --> 00:55:06.640 Okay. s636 00:55:06.640 --> 00:55:14.240 Thank you very much and uh if anyone wants to find me specifically for more questions, feel free to just come up front and ask me or find me by the side s637 00:55:14.240 --> 00:55:15.440 once the next speaker's here. s638 00:55:15.440 --> 00:55:16.680 Yeah. s639 00:55:16.680 --> 00:55:19.000 Thank you very much.