The Frontier AI Inference Cloud for Agents — Byung-Gon (Gon) Chun, FriendliAI
https://www.youtube.com/watch?v=Hvb2LfMH58c
AI Engineer · 14 min · 155 sentences · sentence-level transcript from YouTube's caption track; each line begins at the time shown

[00:13] Let's uh get started.
[00:14] Uh hi everyone.
[00:15] Thank you for coming.
[00:17] Uh this is the late afternoon in the last day.
[00:22] Um so I really appre appreciate it.
[00:25] I'm gone, founder and CEO of friendly AI.
[00:28] Today I want to talk about agentic inference.
[00:31] So I'll first walk through what changed uh why it matters and how we rebuilt the inference cloud for agents.
[00:40] Before we go deeper let me briefly introduce friendly AI.
[00:45] Friendly AI is the frontier AI inference cloud for agents.
[00:49] So we run inference for agents at scale faster, cheaper and more reliably.
[00:55] We are born from a research team at S National University and those research roots still define us.
[01:03] We are the team that invented continuous batching.
[01:06] The inference optimization that is now standard across the industry and our orca work inspired 3LM a widely used open source framework.
[01:16] Today we operate globally headquartered in San Francisco with a team in soul to scale frontier inference.
[01:25] As you know 2026 is a year agents go into massive production and it's driven by two trends coming together.
[01:34] First agents are going exponential.
[01:37] AI agents are driving explosive adoption across software operations and knowledge work.
[01:45] Second, open rate motors have reached the frontier and make agents economic.
[01:51] They now rival closed frontier motors in capability, which means you can run frontier quality agents on open motors with much lower token cost.
[02:02] Let me make the openweight motor part concrete.
[02:06] Open weight motors are now strong enough for these types of real agentic workflows.
[02:11] Here we gave the exact same task building a tower defense game with a coding agent to two models.
[02:19] On the left is GLM 5.2 an openw rate motor running on friendly AI.
[02:24] On the right is anthropics opus 4.8. The important point is not that the outputs are identical.
[02:32] The point is that both complete the task at a level that is clearly usable for many agentic workflows.
[02:41] Open weight models have crossed the quality threshold but the economics are very different for the same task.
[02:51] Opus 4.8 cost about a $150 GLM 5.2 unfriendly AI cost 27 cents about 5.6 times cheaper.
[03:03] So this is the promise I mentioned earlier.
[03:05] Open rate models give you frontier quality agents at a fraction of cost.
[03:12] But motor cost is only one part of the story.
[03:15] To make agent actually fast and reliable, the inference stack itself has to change.
[03:22] So let's look at what actually happens inside an agentic workload.
[03:28] So first let's look at changes in the workload.
[03:31] In the past the dominant usage was chat.
[03:34] The basic unit was a request.
[03:36] A person asks a question, the model answers and the person reads it.
[03:42] Latency meant how fast did I get one response.
[03:46] Agents are different.
[03:48] The basic unit is a task.
[03:50] A task may involve many model cores, many tool cores and it may run autonomously for a while.
[03:58] So the user does not really care about the latency of one individual request.
[04:04] The user cares about when the whole task is completed.
[04:08] That means we have to optimize for tasks, not just individual requests.
[04:15] Let's look at agent workflows more closely.
[04:18] An agent really runs a session made up of tasks.
[04:23] Each task typically runs in a loop.
[04:25] First, it plans, which usually means an LLM call.
[04:29] Then, it acts maybe by calling a tool.
[04:32] Then it observes the result and adds the that back into the context and it repeats this until the task is done.
[04:42] So we are constantly alternating between LLM inference and one or more nonLM tool executions.
[04:50] So there is a gap between LLM calls.
[04:54] An agent can also create sub agents and run them in parallel.
[05:01] Agent inputs also look very different from chat.
[05:04] The graph here shows the prompt and completion length distributions of our internal coding agent runs with GLM 5.2 which we use day-to-day.
[05:16] They are much longer.
[05:18] They grow as the task progresses since every observation gets appended back into the context.
[05:25] There's an important pattern here.
[05:27] Consecutive agent steps usually share a huge prefix.
[05:32] If we recomp compute the same prefix every time, we are burning a lot of compute on work we already did.
[05:40] So this is one of the biggest opportunities in agentic inference.
[05:46] So how token hungry are agents?
[05:49] Now let's look at a long horizon task example like deep research.
[05:54] We ran explained the spec decoding framework in VLM using code with GLM 5.2 unfriendly AI.
[06:03] There are multiple stages and each stage is composed of sub agents which run multiple inferences and tool cores.
[06:12] So it might run tens or even hundreds of inference steps sometimes over minutes or hours and the shared context keeps going the whole time.
[06:23] For the user what matters is not the latency of a single token or one core.
[06:28] What matters is when is my test completed.
[06:35] So agent inference is not just chat with more requests.
[06:39] It's a different problem.
[06:41] The context grows over time.
[06:43] Tour work is interled between model cores.
[06:47] The number of model cores depends on the input.
[06:49] So you can't really plan around a fixed request rate plan.
[06:55] And the real metric is end to end test latency, not a single request latency.
[07:02] This is where friendly AI comes in.
[07:05] We rebuilt the frontier inference cloud specifically for agentic goal close around the challenges I just walked through and we set one goal optimize end to end test latency the task not just the request.
[07:20] So how do we do that?
[07:22] Let me show you the key engineering behind it.
[07:27] Here's the engineering map for how we think about it.
[07:30] We built the stack layer by layer around agent workflows.
[07:34] There are four big pillars I'm going to cover today.
[07:38] Prefix caching key value in short KB cache management, cash aware routing, agent aware optimization and of course underneath we need model layer optimization like sparse attention for long context
[07:56] techniques to reduce errors, fast corners, resilience serving and more.
[08:02] In this talk, I'm going to focus on the four pillars.
[08:08] Let's start with prefix caching.
[08:10] Since Asian steps share a large prefix, we compute key value for that prefix once and cache it.
[08:17] Then on later steps, we reuse the cache key value and only process the new suffix.
[08:24] Reading from cache is much cheaper than recmp computing prefill.
[08:28] So this improves time to first token and reduces compute on every step.
[08:33] And the longer the task runs in agents, the more valuable this becomes.
[08:41] But caching only works if the KV cache actually fits and can move around efficiently.
[08:48] So we need strong KV cache management.
[08:52] We use frugal memory management to pack more active context onto each GPU memory.
[08:59] We use KB contigation to reduce the memory footprint.
[09:05] We you we use hierarchical caching across GPU memory, host memory and disks.
[09:11] So we can go beyond GPU limits.
[09:14] And we also use distributed caching.
[09:17] So one prefix can be served across replicas, not just inside one instance.
[09:26] At global cluster scale, routing becomes really important.
[09:29] A knife load balancer may spread requests evenly across GPU clusters, but it can destroy cache locality.
[09:38] A cache aware router at a global scale does something smarter.
[09:42] It sends a request to part that already has the right prefix cached turning a cord prefill into one cache ship.
[09:51] At the same time, it still has to balance load.
[09:54] So one part doesn't become a hot spot.
[09:58] In this example, the two requests of task A go to the same part one for cache locality.
[10:07] The next piece is agent aware optimization.
[10:11] And this is the next fronture of agentic inference.
[10:16] Today, most systems schedule each LLM core as if it were independent.
[10:21] They don't really understand that this core is part of a longer agent program.
[10:27] But if the optimizer knows the agent level context, we can make better decisions.
[10:34] For example, preempting the right work, speculatively prefilling context for an unlikely next step or making a better cache eviction decision based on agent level context.
[10:49] So the goal is to reduce ant latency not just make one call look fast.
[10:57] When we put all of this together, this is the payoff.
[11:01] We are using the same model GLM 5.2 with kilo code to create a simple mobile game.
[11:08] We ran the same task with model APIs of friendly AI and another well-known inference provider.
[11:15] As you can see, friendly completes the same task end to end tox thanks to our Asenticentric cloud design.
[11:27] So what does this unlock in practice?
[11:29] a stronger production agent stack.
[11:33] Take an agent you already like.
[11:36] Now plug in openweight frontier models like GLM 5.2, Minimax and Kimi served on friendly AI.
[11:44] The motor gives you frontier quality capability and better economics.
[11:50] Friendly AI gives you the speed, reliability and endtoend test performance needed in production.
[11:57] That combination, quality, speed, reliability, and cost is what makes agents actually useful and economical in production.
[12:09] Friendly AI is currently powering teams in production from AI native startups to global enterprises.
[12:16] I'd like to highlight a couple here.
[12:20] Hilo is a hugely popular Asantic AI coding tool serving millions of users.
[12:27] LG is a global enterprise whose businesses range from electronics to healthcare to energy.
[12:35] Very different companies, but they all need the same thing.
[12:39] Fast, reliable, cost effective agentic inference.
[12:45] This testimonial from our client Kilo says it all.
[12:50] Over the past year, Kilo Code has tested several inference providers hosting both open and closed models.
[12:56] In a split test of GLM5 usage compared against other thirdparty providers and direct usage from the model lab G.A.I., Friendly AI was consistently seven times faster with a significantly
[13:11] lower error rate.
[13:13] Today, friendly AI is a core component of the killer stack.
[13:20] And you can consume this however fits your stack.
[13:23] Model API is the fastest way to start.
[13:26] Core frontier open weight models through our serless API.
[13:30] Dedicated endpoints give you your own isolated deployment with guaranteed SLAs's for production workloads.
[13:37] And BYOG bring your own GPU lets you run friendly inference on your own infrastructure.
[13:44] Same stack, three ways to deploy.
[13:49] To wrap up, there are three things to remember.
[13:53] First, frontier open weight models make production agents economically scalable.
[13:59] Second, agents are not just chat with more cores.
[14:03] Agentic inference requires optimizing end to end task latency with the challenges I mentioned.
[14:10] Third, friendly AI is built as an inference cloud for that word.
[14:16] Fast, reliable, cost effective agentic inference.
[14:23] Thank you for attending my session.
[14:25] If you're building agents, give uh frontier openweight motors a try on friendly today.
[14:30] You can get started at friendly.ai in minutes.
[14:34] And uh thank you.
[14:35] I'll be around after the session.
[14:37] Um thank you.
[14:40] [applause]
[14:54] [music]