WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=_naLJOqVOEI NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/_naLJOqVOEI.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:01.309 --> 00:00:03.309 [music] s2 00:00:13.280 --> 00:00:15.120 All right, let's get started. s3 00:00:15.120 --> 00:00:17.039 So, hello everyone. s4 00:00:17.039 --> 00:00:18.400 My name is Yun Mo. s5 00:00:18.400 --> 00:00:25.920 I'm a founding engineer at Friendly AI uh building inference engine with our team to optimize inference performance. s6 00:00:25.920 --> 00:00:34.719 Uh so today I'm going to talk about how we built the fast and reliable inference API for frontier openweight models. s7 00:00:34.960 --> 00:00:39.360 So oh thank you. s8 00:00:39.360 --> 00:00:44.160 Okay so before we dive in let's do a quick show of hands. s9 00:00:44.160 --> 00:00:51.520 So who here uses closed weight frontier models like GPT or cloud? s10 00:00:51.520 --> 00:00:54.239 Okay most of you are already using it. s11 00:00:54.239 --> 00:00:59.120 And then who here mainly use openweight models? s12 00:00:59.920 --> 00:01:00.559 Yeah. s13 00:01:00.559 --> 00:01:01.440 Yeah. s14 00:01:01.440 --> 00:01:03.760 Fewer than the closed weight models. s15 00:01:03.760 --> 00:01:13.360 So personally I recently using a mix of both these days but I'm gradually migrating more of my workload to open weight models. s16 00:01:13.360 --> 00:01:18.880 The reason is that open weight models uh have crossed the threshold these days. s17 00:01:18.880 --> 00:01:22.560 For example, encoding, reasoning, and agentic benchmarks. s18 00:01:22.560 --> 00:01:28.720 Models like GLM 5.2 are competing in the same tier as the leading proprietary models. s19 00:01:28.720 --> 00:01:42.240 So, as you can see from this code arena leaderboard, uh GLM 5.2 is ranked even higher than code opus 4.8. So, it is right uh below the cloud fable 5. s20 00:01:42.240 --> 00:01:51.280 So, similarly across various coding terminal and tool use benchmarks uh GM 5.2 2 and minmax M3 shows strong results. s21 00:01:51.280 --> 00:01:59.360 So the question is no longer only whether open weight models are good enough for many workloads now they are. s22 00:01:59.360 --> 00:02:05.840 So the next question is how fast and reliably we can run them in production. s23 00:02:05.840 --> 00:02:12.720 So benchmark numbers alone uh don't really give a clear sense of how well things actually works. s24 00:02:12.720 --> 00:02:19.200 So instead I ran a a simple hands-on experiment to see how these models actually perform. s25 00:02:19.200 --> 00:02:26.800 So I asked both GLM 5.2 and OPUS 4.8 uh 4.8 to build a tower defense game like this. s26 00:02:26.800 --> 00:02:32.319 So as you can see both models produce working implementation with similar structure and quality. s27 00:02:32.319 --> 00:02:38.800 So both one has a map and the turrets are shooting the the enemies within the range. s28 00:02:38.800 --> 00:02:44.640 So yeah, both looks pretty great considering I made it with a a single prompt. s29 00:02:44.640 --> 00:02:48.319 But when we look at a cost, the difference becomes very clear. s30 00:02:48.319 --> 00:02:54.239 So GM 5.2 was more than 5.6 times cheaper for this task. s31 00:02:54.239 --> 00:03:02.400 And in agent SL style workflows where you need to iterate multiple times, the the the gap compounds quickly. s32 00:03:02.400 --> 00:03:05.519 So this is not just about motor quality anymore. s33 00:03:05.519 --> 00:03:09.519 It is about what you can afford to run at scale. s34 00:03:09.519 --> 00:03:18.480 So after the model quality and cost, the next next question is provider quality and provider quality is really about performance. s35 00:03:18.480 --> 00:03:26.800 So even if the model weights are the same, latency, throughput and reliability can vary significantly depending on where you run it. s36 00:03:26.800 --> 00:03:33.440 Uh so this one has a open router snapshot which shows that uh these differences are measurable. s37 00:03:33.440 --> 00:03:45.519 So if you're building an AI product, you should not only ask which model to use, but you should also ask which providers gives you the best performance, reliability, and pricing. s38 00:03:45.519 --> 00:03:47.680 So I will show you another example. s39 00:03:47.680 --> 00:03:53.599 So here I use the same model 5.2 and the same coding agent workflow. s40 00:03:53.599 --> 00:03:59.519 So the task is to build a simple Candy Crush style mobile game which we call moto crush. s41 00:03:59.519 --> 00:04:02.720 So when you see this, the left side is friendly AI. s42 00:04:02.720 --> 00:04:06.879 it generates the the the game within two minutes. s43 00:04:06.879 --> 00:04:11.200 Another provider needs uh more than three minutes. s44 00:04:11.200 --> 00:04:12.799 So the model is the same. s45 00:04:12.799 --> 00:04:16.639 The agent app is the same but the provider was different. s46 00:04:16.639 --> 00:04:24.800 So model developer tools and coding agent this is this exper this is the experience that users actually feels. s47 00:04:24.800 --> 00:04:34.479 So from this simple example we saw that even with the same model and the same task the experience can feel very different depending on the provider. s48 00:04:34.479 --> 00:04:37.360 The difference comes from the entire inference tech. s49 00:04:37.360 --> 00:04:50.080 So the model architecture cache system routing policy quantization strategy GPU corners all those things come together should shape what the developer actually experiences. s50 00:04:50.080 --> 00:04:53.759 So this is the uh core of friendly AI's value. s51 00:04:53.759 --> 00:05:02.960 So we are not just exposing a model API be uh so friendly AI is built as a fullstack inference engine for geni models. s52 00:05:02.960 --> 00:05:08.240 This matters uh because fast and reliable inference is not just one optimization. s53 00:05:08.240 --> 00:05:18.320 It is batching scheduling quantization spec decoding and connors all all everything should work together. s54 00:05:18.320 --> 00:05:28.880 So let me briefly explain uh how we built that and how we make uh the inference performance uh to the cutting edge level. s55 00:05:28.880 --> 00:05:33.919 So in the modern agent applications inputs are often very long. s56 00:05:33.919 --> 00:05:45.520 It easily exceeds uh 10k tokens while outputs are relatively short and more importantly a large portion of the inputs are repeated across requests. s57 00:05:45.520 --> 00:05:53.520 In many real workloads like 50% to 80 to 90% of the tokens are identical across different requests. s58 00:05:53.520 --> 00:06:04.080 So from a system perspective this means uh most of the compute cost is in the prefill stage where the model processes the entire inputs and build a KB cache. s59 00:06:04.080 --> 00:06:10.479 So if we recompuse this share prefix every time you're wasting both latency and GPU cycles. s60 00:06:10.479 --> 00:06:22.240 So when a new reccast share a a prefix with the previous one, we can reuse those tensor and skip the corresponding prefer computation and directly reduce the TTFT and s61 00:06:22.240 --> 00:06:31.199 uh we can avoid recmp computing large chunk of the input and but however making this work at scale requires two key ch uh mechanisms. s62 00:06:31.199 --> 00:06:33.680 So first one is cache aware routing. s63 00:06:33.680 --> 00:06:37.280 So in the distributed system KB cache is not centralized. s64 00:06:37.280 --> 00:06:39.600 It lives on individual replicas. s65 00:06:39.600 --> 00:06:50.319 So the router must send incoming request to the replica that already holds the relevant prefix in its cache otherwise we re we lose the benefit of reuse. s66 00:06:50.319 --> 00:06:52.960 The second thing is cache hierarchy. s67 00:06:52.960 --> 00:06:58.319 So KB cache is extremely memory intensive for large models and long context. s68 00:06:58.319 --> 00:07:00.960 KB cache size can exceed model weights. s69 00:07:00.960 --> 00:07:06.560 So and as you know GPU VRAM is the fastest but there's also very limited resources. s70 00:07:06.560 --> 00:07:18.560 So uh we treat cache as a hierarchy like hot cache stays in GPU memory warm cachees in uh CPU RAM and quarter cache can be offloaded to MVME or s71 00:07:18.560 --> 00:07:19.919 uh the disk spaces. s72 00:07:19.919 --> 00:07:26.720 The system dynamically offloads less used blocks and prep them when uh when they need it. s73 00:07:26.720 --> 00:07:40.080 So by combining cash aware routing and hierarchical memory management we significantly reduce the prefix cost lower TTFT and making long context agent workload much more efficient. s74 00:07:40.080 --> 00:07:42.880 So the next thing is interactive speed. s75 00:07:42.880 --> 00:07:59.440 So standard auto reggressive decoding it generates one tokens per four pass that makes decode uh inherently a sequenture and during the decode the GPU is often underutilized and speculative decoding changes the shape of this problem. s76 00:07:59.440 --> 00:08:14.240 So instead of generating one token and stopping it proposes multiple candidates tokens ahead then the target model validates those uh candidates in a single pass and if the tokens are accepted they are free. s77 00:08:14.240 --> 00:08:24.639 So this improves time per output token uh without changing models uh output quality and there are different ways to draft candidates. s78 00:08:24.639 --> 00:08:28.160 So one way is modelbased speculation. s79 00:08:28.160 --> 00:08:40.320 It uses a small draft model to predict the future tokens and then and the benefit of it is it can achieve higher acceptance rate but the cost is the s80 00:08:40.320 --> 00:08:45.600 uh extra compute and another method is model free speculation. s81 00:08:45.600 --> 00:08:49.279 It uses pattern from a prompt or prior output. s82 00:08:49.279 --> 00:08:56.720 So it has lower overhead but it works best only when the output resembles the existing context. s83 00:08:56.720 --> 00:08:59.360 So the important part is the trade-off. s84 00:08:59.360 --> 00:09:05.760 So speculation can improve per request latency but validation as overhead to the batch. s85 00:09:05.760 --> 00:09:08.080 So the best strategy is not static. s86 00:09:08.080 --> 00:09:12.160 And at friendly AI we take a hybrid approach. s87 00:09:12.160 --> 00:09:21.120 So the runtime can dynamically select between model based and model speculation uh based on the workload and request uh characteristic. s88 00:09:21.120 --> 00:09:26.160 So we also think about speculation as something we can improve over time. s89 00:09:26.160 --> 00:09:32.640 So with self-evolving speculation uh draft model can adapt based on the real serving traffic. s90 00:09:32.640 --> 00:09:42.480 So the speculator continuously improves its prediction of the workload actually running in the in your uh production. s91 00:09:42.480 --> 00:09:46.160 So the next thing I want to mention is reliability. s92 00:09:46.160 --> 00:09:49.040 So speed alone is not enough. s93 00:09:49.040 --> 00:09:53.839 So production AI application uh reliability is part of speed. s94 00:09:53.839 --> 00:10:07.440 So if an API is fast in average but frequently hits timeout like five 5xx error 4 to9 rate limit or fail or that kind of failure the user experiences s95 00:10:07.440 --> 00:10:08.800 uh still breaks. s96 00:10:08.800 --> 00:10:14.000 So this becomes even more important for agentic workloads. s97 00:10:14.000 --> 00:10:24.000 As you know agent may need multiple model cores to complete one task and if just one call phase in the middle the entire workflow can be invalidated. s98 00:10:24.000 --> 00:10:27.920 So that's why we do not optimize only for the latency or throughput. s99 00:10:27.920 --> 00:10:34.399 We optimize the full API experiences uh including the error rate, uptime, all those things together. s100 00:10:34.399 --> 00:10:49.279 And this open AAI snapshot for GLM 5.1 pro uh shows that friendly AI is not only the fast but it's also highly reliable with over 99.9% up time. s101 00:10:49.279 --> 00:11:00.079 So what I want to say is many inference provider out here of course in this menu as well they will tell you that they are fast and some point of them s102 00:11:00.079 --> 00:11:10.880 they are but on cert on certain benchmark under certain condition but for your application the only benchmark that really matters is your own workload. s103 00:11:10.880 --> 00:11:17.360 So your prompt your token length your traffic patterns and your latency expectation. s104 00:11:17.360 --> 00:11:20.320 So our recommendation is simple. s105 00:11:20.320 --> 00:11:29.519 So please bring your workload to us and test a poor experience with us in terms of like speed, reliability, cost and quality. s106 00:11:29.519 --> 00:11:38.800 And of course um we build our inference service for exactly these production workloads and we and we believe that we can give you the best experience. s107 00:11:38.800 --> 00:11:40.880 So we have a boost in this venue. s108 00:11:40.880 --> 00:11:45.279 So bring your hardest problem to us and let's solve it together. s109 00:11:45.279 --> 00:11:51.920 So to close I want to share a testimonial from one of our customer is a kilo code. s110 00:11:51.920 --> 00:12:00.000 So kilo code is one of the largest coding agent platform processing like several trillions of tokens every week. s111 00:12:00.000 --> 00:12:11.360 So they have tested many different inference providers over past years and in their split test for GLM models they were especially impressed by friendly AI. s112 00:12:11.360 --> 00:12:20.560 So compared with many other inference provider, friendly AI was over like seven times faster and had a sign significantly lower error rate. s113 00:12:20.560 --> 00:12:24.720 So I believe you can get the same kind of experience with friendly AI. s114 00:12:24.720 --> 00:12:37.120 So if you're building your frontier open models, building with your frontier open models, come try them on friendly AI and bring your real workload and let us show you s115 00:12:37.120 --> 00:12:40.399 what fast and reliable inference can do. s116 00:12:40.399 --> 00:12:45.200 So this is the end of my talk and if you have any question feel free to ask. s117 00:12:45.200 --> 00:12:47.440 Thank you. s118 00:13:02.078 --> 00:13:04.078 [music]