Same Model, Different Speed: Why Your Inference Provider Matters — FriendliAI
https://www.youtube.com/watch?v=_naLJOqVOEI
AI Engineer · 13 min · 118 sentences · sentence-level transcript from YouTube's caption track; each line begins at the time shown

[00:01] [music]
[00:13] All right, let's get started.
[00:15] So, hello everyone.
[00:17] My name is Yun Mo.
[00:18] I'm a founding engineer at Friendly AI uh building inference engine with our team to optimize inference performance.
[00:25] Uh so today I'm going to talk about how we built the fast and reliable inference API for frontier openweight models.
[00:34] So oh thank you.
[00:39] Okay so before we dive in let's do a quick show of hands.
[00:44] So who here uses closed weight frontier models like GPT or cloud?
[00:51] Okay most of you are already using it.
[00:54] And then who here mainly use openweight models?
[00:59] Yeah.
[01:00] Yeah.
[01:01] Fewer than the closed weight models.
[01:03] So personally I recently using a mix of both these days but I'm gradually migrating more of my workload to open weight models.
[01:13] The reason is that open weight models uh have crossed the threshold these days.
[01:18] For example, encoding, reasoning, and agentic benchmarks.
[01:22] Models like GLM 5.2 are competing in the same tier as the leading proprietary models.
[01:28] So, as you can see from this code arena leaderboard, uh GLM 5.2 is ranked even higher than code opus 4.8. So, it is right uh below the cloud fable 5.
[01:42] So, similarly across various coding terminal and tool use benchmarks uh GM 5.2 2 and minmax M3 shows strong results.
[01:51] So the question is no longer only whether open weight models are good enough for many workloads now they are.
[01:59] So the next question is how fast and reliably we can run them in production.
[02:05] So benchmark numbers alone uh don't really give a clear sense of how well things actually works.
[02:12] So instead I ran a a simple hands-on experiment to see how these models actually perform.
[02:19] So I asked both GLM 5.2 and OPUS 4.8 uh 4.8 to build a tower defense game like this.
[02:26] So as you can see both models produce working implementation with similar structure and quality.
[02:32] So both one has a map and the turrets are shooting the the enemies within the range.
[02:38] So yeah, both looks pretty great considering I made it with a a single prompt.
[02:44] But when we look at a cost, the difference becomes very clear.
[02:48] So GM 5.2 was more than 5.6 times cheaper for this task.
[02:54] And in agent SL style workflows where you need to iterate multiple times, the the the gap compounds quickly.
[03:02] So this is not just about motor quality anymore.
[03:05] It is about what you can afford to run at scale.
[03:09] So after the model quality and cost, the next next question is provider quality and provider quality is really about performance.
[03:18] So even if the model weights are the same, latency, throughput and reliability can vary significantly depending on where you run it.
[03:26] Uh so this one has a open router snapshot which shows that uh these differences are measurable.
[03:33] So if you're building an AI product, you should not only ask which model to use, but you should also ask which providers gives you the best performance, reliability, and pricing.
[03:45] So I will show you another example.
[03:47] So here I use the same model 5.2 and the same coding agent workflow.
[03:53] So the task is to build a simple Candy Crush style mobile game which we call moto crush.
[03:59] So when you see this, the left side is friendly AI.
[04:02] it generates the the the game within two minutes.
[04:06] Another provider needs uh more than three minutes.
[04:11] So the model is the same.
[04:12] The agent app is the same but the provider was different.
[04:16] So model developer tools and coding agent this is this exper this is the experience that users actually feels.
[04:24] So from this simple example we saw that even with the same model and the same task the experience can feel very different depending on the provider.
[04:34] The difference comes from the entire inference tech.
[04:37] So the model architecture cache system routing policy quantization strategy GPU corners all those things come together should shape what the developer actually experiences.
[04:50] So this is the uh core of friendly AI's value.
[04:53] So we are not just exposing a model API be uh so friendly AI is built as a fullstack inference engine for geni models.
[05:02] This matters uh because fast and reliable inference is not just one optimization.
[05:08] It is batching scheduling quantization spec decoding and connors all all everything should work together.
[05:18] So let me briefly explain uh how we built that and how we make uh the inference performance uh to the cutting edge level.
[05:28] So in the modern agent applications inputs are often very long.
[05:33] It easily exceeds uh 10k tokens while outputs are relatively short and more importantly a large portion of the inputs are repeated across requests.
[05:45] In many real workloads like 50% to 80 to 90% of the tokens are identical across different requests.
[05:53] So from a system perspective this means uh most of the compute cost is in the prefill stage where the model processes the entire inputs and build a KB cache.
[06:04] So if we recompuse this share prefix every time you're wasting both latency and GPU cycles.
[06:10] So when a new reccast share a a prefix with the previous one, we can reuse those tensor and skip the corresponding prefer computation and directly reduce the TTFT and
[06:22] uh we can avoid recmp computing large chunk of the input and but however making this work at scale requires two key ch uh mechanisms.
[06:31] So first one is cache aware routing.
[06:33] So in the distributed system KB cache is not centralized.
[06:37] It lives on individual replicas.
[06:39] So the router must send incoming request to the replica that already holds the relevant prefix in its cache otherwise we re we lose the benefit of reuse.
[06:50] The second thing is cache hierarchy.
[06:52] So KB cache is extremely memory intensive for large models and long context.
[06:58] KB cache size can exceed model weights.
[07:00] So and as you know GPU VRAM is the fastest but there's also very limited resources.
[07:06] So uh we treat cache as a hierarchy like hot cache stays in GPU memory warm cachees in uh CPU RAM and quarter cache can be offloaded to MVME or
[07:18] uh the disk spaces.
[07:19] The system dynamically offloads less used blocks and prep them when uh when they need it.
[07:26] So by combining cash aware routing and hierarchical memory management we significantly reduce the prefix cost lower TTFT and making long context agent workload much more efficient.
[07:40] So the next thing is interactive speed.
[07:42] So standard auto reggressive decoding it generates one tokens per four pass that makes decode uh inherently a sequenture and during the decode the GPU is often underutilized and speculative decoding changes the shape of this problem.
[07:59] So instead of generating one token and stopping it proposes multiple candidates tokens ahead then the target model validates those uh candidates in a single pass and if the tokens are accepted they are free.
[08:14] So this improves time per output token uh without changing models uh output quality and there are different ways to draft candidates.
[08:24] So one way is modelbased speculation.
[08:28] It uses a small draft model to predict the future tokens and then and the benefit of it is it can achieve higher acceptance rate but the cost is the
[08:40] uh extra compute and another method is model free speculation.
[08:45] It uses pattern from a prompt or prior output.
[08:49] So it has lower overhead but it works best only when the output resembles the existing context.
[08:56] So the important part is the trade-off.
[08:59] So speculation can improve per request latency but validation as overhead to the batch.
[09:05] So the best strategy is not static.
[09:08] And at friendly AI we take a hybrid approach.
[09:12] So the runtime can dynamically select between model based and model speculation uh based on the workload and request uh characteristic.
[09:21] So we also think about speculation as something we can improve over time.
[09:26] So with self-evolving speculation uh draft model can adapt based on the real serving traffic.
[09:32] So the speculator continuously improves its prediction of the workload actually running in the in your uh production.
[09:42] So the next thing I want to mention is reliability.
[09:46] So speed alone is not enough.
[09:49] So production AI application uh reliability is part of speed.
[09:53] So if an API is fast in average but frequently hits timeout like five 5xx error 4 to9 rate limit or fail or that kind of failure the user experiences
[10:07] uh still breaks.
[10:08] So this becomes even more important for agentic workloads.
[10:14] As you know agent may need multiple model cores to complete one task and if just one call phase in the middle the entire workflow can be invalidated.
[10:24] So that's why we do not optimize only for the latency or throughput.
[10:27] We optimize the full API experiences uh including the error rate, uptime, all those things together.
[10:34] And this open AAI snapshot for GLM 5.1 pro uh shows that friendly AI is not only the fast but it's also highly reliable with over 99.9% up time.
[10:49] So what I want to say is many inference provider out here of course in this menu as well they will tell you that they are fast and some point of them
[11:00] they are but on cert on certain benchmark under certain condition but for your application the only benchmark that really matters is your own workload.
[11:10] So your prompt your token length your traffic patterns and your latency expectation.
[11:17] So our recommendation is simple.
[11:20] So please bring your workload to us and test a poor experience with us in terms of like speed, reliability, cost and quality.
[11:29] And of course um we build our inference service for exactly these production workloads and we and we believe that we can give you the best experience.
[11:38] So we have a boost in this venue.
[11:40] So bring your hardest problem to us and let's solve it together.
[11:45] So to close I want to share a testimonial from one of our customer is a kilo code.
[11:51] So kilo code is one of the largest coding agent platform processing like several trillions of tokens every week.
[12:00] So they have tested many different inference providers over past years and in their split test for GLM models they were especially impressed by friendly AI.
[12:11] So compared with many other inference provider, friendly AI was over like seven times faster and had a sign significantly lower error rate.
[12:20] So I believe you can get the same kind of experience with friendly AI.
[12:24] So if you're building your frontier open models, building with your frontier open models, come try them on friendly AI and bring your real workload and let us show you
[12:37] what fast and reliable inference can do.
[12:40] So this is the end of my talk and if you have any question feel free to ask.
[12:45] Thank you.
[13:02] [music]