Same Model, Different Speed: Why Your Inference Provider Matters — FriendliAI

AI Engineer · 13 min · 118 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

1. 00:01[music]
2. 00:13All right, let's get started.
3. 00:15So, hello everyone.
4. 00:17My name is Yun Mo.
5. 00:18I'm a founding engineer at Friendly AI uh building inference engine with our team to optimize inference performance.
6. 00:25Uh so today I'm going to talk about how we built the fast and reliable inference API for frontier openweight models.
7. 00:34So oh thank you.
8. 00:39Okay so before we dive in let's do a quick show of hands.
9. 00:44So who here uses closed weight frontier models like GPT or cloud?
10. 00:51Okay most of you are already using it.
11. 00:54And then who here mainly use openweight models?
12. 00:59Yeah.
13. 01:00Yeah.
14. 01:01Fewer than the closed weight models.
15. 01:03So personally I recently using a mix of both these days but I'm gradually migrating more of my workload to open weight models.
16. 01:13The reason is that open weight models uh have crossed the threshold these days.
17. 01:18For example, encoding, reasoning, and agentic benchmarks.
18. 01:22Models like GLM 5.2 are competing in the same tier as the leading proprietary models.
19. 01:28So, as you can see from this code arena leaderboard, uh GLM 5.2 is ranked even higher than code opus 4.8. So, it is right uh below the cloud fable 5.
20. 01:42So, similarly across various coding terminal and tool use benchmarks uh GM 5.2 2 and minmax M3 shows strong results.
21. 01:51So the question is no longer only whether open weight models are good enough for many workloads now they are.
22. 01:59So the next question is how fast and reliably we can run them in production.
23. 02:05So benchmark numbers alone uh don't really give a clear sense of how well things actually works.
24. 02:12So instead I ran a a simple hands-on experiment to see how these models actually perform.
25. 02:19So I asked both GLM 5.2 and OPUS 4.8 uh 4.8 to build a tower defense game like this.
26. 02:26So as you can see both models produce working implementation with similar structure and quality.
27. 02:32So both one has a map and the turrets are shooting the the enemies within the range.
28. 02:38So yeah, both looks pretty great considering I made it with a a single prompt.
29. 02:44But when we look at a cost, the difference becomes very clear.
30. 02:48So GM 5.2 was more than 5.6 times cheaper for this task.
31. 02:54And in agent SL style workflows where you need to iterate multiple times, the the the gap compounds quickly.
32. 03:02So this is not just about motor quality anymore.
33. 03:05It is about what you can afford to run at scale.
34. 03:09So after the model quality and cost, the next next question is provider quality and provider quality is really about performance.
35. 03:18So even if the model weights are the same, latency, throughput and reliability can vary significantly depending on where you run it.
36. 03:26Uh so this one has a open router snapshot which shows that uh these differences are measurable.
37. 03:33So if you're building an AI product, you should not only ask which model to use, but you should also ask which providers gives you the best performance, reliability, and pricing.
38. 03:45So I will show you another example.
39. 03:47So here I use the same model 5.2 and the same coding agent workflow.
40. 03:53So the task is to build a simple Candy Crush style mobile game which we call moto crush.
41. 03:59So when you see this, the left side is friendly AI.
42. 04:02it generates the the the game within two minutes.
43. 04:06Another provider needs uh more than three minutes.
44. 04:11So the model is the same.
45. 04:12The agent app is the same but the provider was different.
46. 04:16So model developer tools and coding agent this is this exper this is the experience that users actually feels.
47. 04:24So from this simple example we saw that even with the same model and the same task the experience can feel very different depending on the provider.
48. 04:34The difference comes from the entire inference tech.
49. 04:37So the model architecture cache system routing policy quantization strategy GPU corners all those things come together should shape what the developer actually experiences.
50. 04:50So this is the uh core of friendly AI's value.
51. 04:53So we are not just exposing a model API be uh so friendly AI is built as a fullstack inference engine for geni models.
52. 05:02This matters uh because fast and reliable inference is not just one optimization.
53. 05:08It is batching scheduling quantization spec decoding and connors all all everything should work together.
54. 05:18So let me briefly explain uh how we built that and how we make uh the inference performance uh to the cutting edge level.
55. 05:28So in the modern agent applications inputs are often very long.
56. 05:33It easily exceeds uh 10k tokens while outputs are relatively short and more importantly a large portion of the inputs are repeated across requests.
57. 05:45In many real workloads like 50% to 80 to 90% of the tokens are identical across different requests.
58. 05:53So from a system perspective this means uh most of the compute cost is in the prefill stage where the model processes the entire inputs and build a KB cache.
59. 06:04So if we recompuse this share prefix every time you're wasting both latency and GPU cycles.
60. 06:10So when a new reccast share a a prefix with the previous one, we can reuse those tensor and skip the corresponding prefer computation and directly reduce the TTFT and
61. 06:22uh we can avoid recmp computing large chunk of the input and but however making this work at scale requires two key ch uh mechanisms.
62. 06:31So first one is cache aware routing.
63. 06:33So in the distributed system KB cache is not centralized.
64. 06:37It lives on individual replicas.
65. 06:39So the router must send incoming request to the replica that already holds the relevant prefix in its cache otherwise we re we lose the benefit of reuse.
66. 06:50The second thing is cache hierarchy.
67. 06:52So KB cache is extremely memory intensive for large models and long context.
68. 06:58KB cache size can exceed model weights.
69. 07:00So and as you know GPU VRAM is the fastest but there's also very limited resources.
70. 07:06So uh we treat cache as a hierarchy like hot cache stays in GPU memory warm cachees in uh CPU RAM and quarter cache can be offloaded to MVME or
71. 07:18uh the disk spaces.
72. 07:19The system dynamically offloads less used blocks and prep them when uh when they need it.
73. 07:26So by combining cash aware routing and hierarchical memory management we significantly reduce the prefix cost lower TTFT and making long context agent workload much more efficient.
74. 07:40So the next thing is interactive speed.
75. 07:42So standard auto reggressive decoding it generates one tokens per four pass that makes decode uh inherently a sequenture and during the decode the GPU is often underutilized and speculative decoding changes the shape of this problem.
76. 07:59So instead of generating one token and stopping it proposes multiple candidates tokens ahead then the target model validates those uh candidates in a single pass and if the tokens are accepted they are free.
77. 08:14So this improves time per output token uh without changing models uh output quality and there are different ways to draft candidates.
78. 08:24So one way is modelbased speculation.
79. 08:28It uses a small draft model to predict the future tokens and then and the benefit of it is it can achieve higher acceptance rate but the cost is the
80. 08:40uh extra compute and another method is model free speculation.
81. 08:45It uses pattern from a prompt or prior output.
82. 08:49So it has lower overhead but it works best only when the output resembles the existing context.
83. 08:56So the important part is the trade-off.
84. 08:59So speculation can improve per request latency but validation as overhead to the batch.
85. 09:05So the best strategy is not static.
86. 09:08And at friendly AI we take a hybrid approach.
87. 09:12So the runtime can dynamically select between model based and model speculation uh based on the workload and request uh characteristic.
88. 09:21So we also think about speculation as something we can improve over time.
89. 09:26So with self-evolving speculation uh draft model can adapt based on the real serving traffic.
90. 09:32So the speculator continuously improves its prediction of the workload actually running in the in your uh production.
91. 09:42So the next thing I want to mention is reliability.
92. 09:46So speed alone is not enough.
93. 09:49So production AI application uh reliability is part of speed.
94. 09:53So if an API is fast in average but frequently hits timeout like five 5xx error 4 to9 rate limit or fail or that kind of failure the user experiences
95. 10:07uh still breaks.
96. 10:08So this becomes even more important for agentic workloads.
97. 10:14As you know agent may need multiple model cores to complete one task and if just one call phase in the middle the entire workflow can be invalidated.
98. 10:24So that's why we do not optimize only for the latency or throughput.
99. 10:27We optimize the full API experiences uh including the error rate, uptime, all those things together.
100. 10:34And this open AAI snapshot for GLM 5.1 pro uh shows that friendly AI is not only the fast but it's also highly reliable with over 99.9% up time.
101. 10:49So what I want to say is many inference provider out here of course in this menu as well they will tell you that they are fast and some point of them
102. 11:00they are but on cert on certain benchmark under certain condition but for your application the only benchmark that really matters is your own workload.
103. 11:10So your prompt your token length your traffic patterns and your latency expectation.
104. 11:17So our recommendation is simple.
105. 11:20So please bring your workload to us and test a poor experience with us in terms of like speed, reliability, cost and quality.
106. 11:29And of course um we build our inference service for exactly these production workloads and we and we believe that we can give you the best experience.
107. 11:38So we have a boost in this venue.
108. 11:40So bring your hardest problem to us and let's solve it together.
109. 11:45So to close I want to share a testimonial from one of our customer is a kilo code.
110. 11:51So kilo code is one of the largest coding agent platform processing like several trillions of tokens every week.
111. 12:00So they have tested many different inference providers over past years and in their split test for GLM models they were especially impressed by friendly AI.
112. 12:11So compared with many other inference provider, friendly AI was over like seven times faster and had a sign significantly lower error rate.
113. 12:20So I believe you can get the same kind of experience with friendly AI.
114. 12:24So if you're building your frontier open models, building with your frontier open models, come try them on friendly AI and bring your real workload and let us show you
115. 12:37what fast and reliable inference can do.
116. 12:40So this is the end of my talk and if you have any question feel free to ask.
117. 12:45Thank you.
118. 13:02[music]