The Frontier AI Inference Cloud for Agents — Byung-Gon (Gon) Chun, FriendliAI

AI Engineer · 14 min · 155 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

1. 00:13Let's uh get started.
2. 00:14Uh hi everyone.
3. 00:15Thank you for coming.
4. 00:17Uh this is the late afternoon in the last day.
5. 00:22Um so I really appre appreciate it.
6. 00:25I'm gone, founder and CEO of friendly AI.
7. 00:28Today I want to talk about agentic inference.
8. 00:31So I'll first walk through what changed uh why it matters and how we rebuilt the inference cloud for agents.
9. 00:40Before we go deeper let me briefly introduce friendly AI.
10. 00:45Friendly AI is the frontier AI inference cloud for agents.
11. 00:49So we run inference for agents at scale faster, cheaper and more reliably.
12. 00:55We are born from a research team at S National University and those research roots still define us.
13. 01:03We are the team that invented continuous batching.
14. 01:06The inference optimization that is now standard across the industry and our orca work inspired 3LM a widely used open source framework.
15. 01:16Today we operate globally headquartered in San Francisco with a team in soul to scale frontier inference.
16. 01:25As you know 2026 is a year agents go into massive production and it's driven by two trends coming together.
17. 01:34First agents are going exponential.
18. 01:37AI agents are driving explosive adoption across software operations and knowledge work.
19. 01:45Second, open rate motors have reached the frontier and make agents economic.
20. 01:51They now rival closed frontier motors in capability, which means you can run frontier quality agents on open motors with much lower token cost.
21. 02:02Let me make the openweight motor part concrete.
22. 02:06Open weight motors are now strong enough for these types of real agentic workflows.
23. 02:11Here we gave the exact same task building a tower defense game with a coding agent to two models.
24. 02:19On the left is GLM 5.2 an openw rate motor running on friendly AI.
25. 02:24On the right is anthropics opus 4.8. The important point is not that the outputs are identical.
26. 02:32The point is that both complete the task at a level that is clearly usable for many agentic workflows.
27. 02:41Open weight models have crossed the quality threshold but the economics are very different for the same task.
28. 02:51Opus 4.8 cost about a $150 GLM 5.2 unfriendly AI cost 27 cents about 5.6 times cheaper.
29. 03:03So this is the promise I mentioned earlier.
30. 03:05Open rate models give you frontier quality agents at a fraction of cost.
31. 03:12But motor cost is only one part of the story.
32. 03:15To make agent actually fast and reliable, the inference stack itself has to change.
33. 03:22So let's look at what actually happens inside an agentic workload.
34. 03:28So first let's look at changes in the workload.
35. 03:31In the past the dominant usage was chat.
36. 03:34The basic unit was a request.
37. 03:36A person asks a question, the model answers and the person reads it.
38. 03:42Latency meant how fast did I get one response.
39. 03:46Agents are different.
40. 03:48The basic unit is a task.
41. 03:50A task may involve many model cores, many tool cores and it may run autonomously for a while.
42. 03:58So the user does not really care about the latency of one individual request.
43. 04:04The user cares about when the whole task is completed.
44. 04:08That means we have to optimize for tasks, not just individual requests.
45. 04:15Let's look at agent workflows more closely.
46. 04:18An agent really runs a session made up of tasks.
47. 04:23Each task typically runs in a loop.
48. 04:25First, it plans, which usually means an LLM call.
49. 04:29Then, it acts maybe by calling a tool.
50. 04:32Then it observes the result and adds the that back into the context and it repeats this until the task is done.
51. 04:42So we are constantly alternating between LLM inference and one or more nonLM tool executions.
52. 04:50So there is a gap between LLM calls.
53. 04:54An agent can also create sub agents and run them in parallel.
54. 05:01Agent inputs also look very different from chat.
55. 05:04The graph here shows the prompt and completion length distributions of our internal coding agent runs with GLM 5.2 which we use day-to-day.
56. 05:16They are much longer.
57. 05:18They grow as the task progresses since every observation gets appended back into the context.
58. 05:25There's an important pattern here.
59. 05:27Consecutive agent steps usually share a huge prefix.
60. 05:32If we recomp compute the same prefix every time, we are burning a lot of compute on work we already did.
61. 05:40So this is one of the biggest opportunities in agentic inference.
62. 05:46So how token hungry are agents?
63. 05:49Now let's look at a long horizon task example like deep research.
64. 05:54We ran explained the spec decoding framework in VLM using code with GLM 5.2 unfriendly AI.
65. 06:03There are multiple stages and each stage is composed of sub agents which run multiple inferences and tool cores.
66. 06:12So it might run tens or even hundreds of inference steps sometimes over minutes or hours and the shared context keeps going the whole time.
67. 06:23For the user what matters is not the latency of a single token or one core.
68. 06:28What matters is when is my test completed.
69. 06:35So agent inference is not just chat with more requests.
70. 06:39It's a different problem.
71. 06:41The context grows over time.
72. 06:43Tour work is interled between model cores.
73. 06:47The number of model cores depends on the input.
74. 06:49So you can't really plan around a fixed request rate plan.
75. 06:55And the real metric is end to end test latency, not a single request latency.
76. 07:02This is where friendly AI comes in.
77. 07:05We rebuilt the frontier inference cloud specifically for agentic goal close around the challenges I just walked through and we set one goal optimize end to end test latency the task not just the request.
78. 07:20So how do we do that?
79. 07:22Let me show you the key engineering behind it.
80. 07:27Here's the engineering map for how we think about it.
81. 07:30We built the stack layer by layer around agent workflows.
82. 07:34There are four big pillars I'm going to cover today.
83. 07:38Prefix caching key value in short KB cache management, cash aware routing, agent aware optimization and of course underneath we need model layer optimization like sparse attention for long context
84. 07:56techniques to reduce errors, fast corners, resilience serving and more.
85. 08:02In this talk, I'm going to focus on the four pillars.
86. 08:08Let's start with prefix caching.
87. 08:10Since Asian steps share a large prefix, we compute key value for that prefix once and cache it.
88. 08:17Then on later steps, we reuse the cache key value and only process the new suffix.
89. 08:24Reading from cache is much cheaper than recmp computing prefill.
90. 08:28So this improves time to first token and reduces compute on every step.
91. 08:33And the longer the task runs in agents, the more valuable this becomes.
92. 08:41But caching only works if the KV cache actually fits and can move around efficiently.
93. 08:48So we need strong KV cache management.
94. 08:52We use frugal memory management to pack more active context onto each GPU memory.
95. 08:59We use KB contigation to reduce the memory footprint.
96. 09:05We you we use hierarchical caching across GPU memory, host memory and disks.
97. 09:11So we can go beyond GPU limits.
98. 09:14And we also use distributed caching.
99. 09:17So one prefix can be served across replicas, not just inside one instance.
100. 09:26At global cluster scale, routing becomes really important.
101. 09:29A knife load balancer may spread requests evenly across GPU clusters, but it can destroy cache locality.
102. 09:38A cache aware router at a global scale does something smarter.
103. 09:42It sends a request to part that already has the right prefix cached turning a cord prefill into one cache ship.
104. 09:51At the same time, it still has to balance load.
105. 09:54So one part doesn't become a hot spot.
106. 09:58In this example, the two requests of task A go to the same part one for cache locality.
107. 10:07The next piece is agent aware optimization.
108. 10:11And this is the next fronture of agentic inference.
109. 10:16Today, most systems schedule each LLM core as if it were independent.
110. 10:21They don't really understand that this core is part of a longer agent program.
111. 10:27But if the optimizer knows the agent level context, we can make better decisions.
112. 10:34For example, preempting the right work, speculatively prefilling context for an unlikely next step or making a better cache eviction decision based on agent level context.
113. 10:49So the goal is to reduce ant latency not just make one call look fast.
114. 10:57When we put all of this together, this is the payoff.
115. 11:01We are using the same model GLM 5.2 with kilo code to create a simple mobile game.
116. 11:08We ran the same task with model APIs of friendly AI and another well-known inference provider.
117. 11:15As you can see, friendly completes the same task end to end tox thanks to our Asenticentric cloud design.
118. 11:27So what does this unlock in practice?
119. 11:29a stronger production agent stack.
120. 11:33Take an agent you already like.
121. 11:36Now plug in openweight frontier models like GLM 5.2, Minimax and Kimi served on friendly AI.
122. 11:44The motor gives you frontier quality capability and better economics.
123. 11:50Friendly AI gives you the speed, reliability and endtoend test performance needed in production.
124. 11:57That combination, quality, speed, reliability, and cost is what makes agents actually useful and economical in production.
125. 12:09Friendly AI is currently powering teams in production from AI native startups to global enterprises.
126. 12:16I'd like to highlight a couple here.
127. 12:20Hilo is a hugely popular Asantic AI coding tool serving millions of users.
128. 12:27LG is a global enterprise whose businesses range from electronics to healthcare to energy.
129. 12:35Very different companies, but they all need the same thing.
130. 12:39Fast, reliable, cost effective agentic inference.
131. 12:45This testimonial from our client Kilo says it all.
132. 12:50Over the past year, Kilo Code has tested several inference providers hosting both open and closed models.
133. 12:56In a split test of GLM5 usage compared against other thirdparty providers and direct usage from the model lab G.A.I., Friendly AI was consistently seven times faster with a significantly
134. 13:11lower error rate.
135. 13:13Today, friendly AI is a core component of the killer stack.
136. 13:20And you can consume this however fits your stack.
137. 13:23Model API is the fastest way to start.
138. 13:26Core frontier open weight models through our serless API.
139. 13:30Dedicated endpoints give you your own isolated deployment with guaranteed SLAs's for production workloads.
140. 13:37And BYOG bring your own GPU lets you run friendly inference on your own infrastructure.
141. 13:44Same stack, three ways to deploy.
142. 13:49To wrap up, there are three things to remember.
143. 13:53First, frontier open weight models make production agents economically scalable.
144. 13:59Second, agents are not just chat with more cores.
145. 14:03Agentic inference requires optimizing end to end task latency with the challenges I mentioned.
146. 14:10Third, friendly AI is built as an inference cloud for that word.
147. 14:16Fast, reliable, cost effective agentic inference.
148. 14:23Thank you for attending my session.
149. 14:25If you're building agents, give uh frontier openweight motors a try on friendly today.
150. 14:30You can get started at friendly.ai in minutes.
151. 14:34And uh thank you.
152. 14:35I'll be around after the session.
153. 14:37Um thank you.
154. 14:40[applause]
155. 14:54[music]