WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=Hvb2LfMH58c

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/Hvb2LfMH58c.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:13.040 --> 00:00:14.719
Let's uh get started.

s2
00:00:14.719 --> 00:00:15.839
Uh hi everyone.

s3
00:00:15.839 --> 00:00:17.279
Thank you for coming.

s4
00:00:17.279 --> 00:00:22.000
Uh this is the late afternoon in the last day.

s5
00:00:22.000 --> 00:00:25.199
Um so I really appre appreciate it.

s6
00:00:25.199 --> 00:00:28.400
I'm gone, founder and CEO of friendly AI.

s7
00:00:28.400 --> 00:00:31.119
Today I want to talk about agentic inference.

s8
00:00:31.119 --> 00:00:40.079
So I'll first walk through what changed uh why it matters and how we rebuilt the inference cloud for agents.

s9
00:00:40.960 --> 00:00:45.600
Before we go deeper let me briefly introduce friendly AI.

s10
00:00:45.600 --> 00:00:49.280
Friendly AI is the frontier AI inference cloud for agents.

s11
00:00:49.280 --> 00:00:55.039
So we run inference for agents at scale faster, cheaper and more reliably.

s12
00:00:55.039 --> 00:01:03.039
We are born from a research team at S National University and those research roots still define us.

s13
00:01:03.039 --> 00:01:06.479
We are the team that invented continuous batching.

s14
00:01:06.479 --> 00:01:16.960
The inference optimization that is now standard across the industry and our orca work inspired 3LM a widely used open source framework.

s15
00:01:16.960 --> 00:01:24.960
Today we operate globally headquartered in San Francisco with a team in soul to scale frontier inference.

s16
00:01:25.360 --> 00:01:34.479
As you know 2026 is a year agents go into massive production and it's driven by two trends coming together.

s17
00:01:34.479 --> 00:01:37.200
First agents are going exponential.

s18
00:01:37.200 --> 00:01:45.040
AI agents are driving explosive adoption across software operations and knowledge work.

s19
00:01:45.040 --> 00:01:51.360
Second, open rate motors have reached the frontier and make agents economic.

s20
00:01:51.360 --> 00:02:02.079
They now rival closed frontier motors in capability, which means you can run frontier quality agents on open motors with much lower token cost.

s21
00:02:02.719 --> 00:02:06.240
Let me make the openweight motor part concrete.

s22
00:02:06.240 --> 00:02:11.599
Open weight motors are now strong enough for these types of real agentic workflows.

s23
00:02:11.599 --> 00:02:19.440
Here we gave the exact same task building a tower defense game with a coding agent to two models.

s24
00:02:19.440 --> 00:02:24.720
On the left is GLM 5.2 an openw rate motor running on friendly AI.

s25
00:02:24.720 --> 00:02:32.959
On the right is anthropics opus 4.8. The important point is not that the outputs are identical.

s26
00:02:32.959 --> 00:02:41.040
The point is that both complete the task at a level that is clearly usable for many agentic workflows.

s27
00:02:41.040 --> 00:02:51.360
Open weight models have crossed the quality threshold but the economics are very different for the same task.

s28
00:02:51.360 --> 00:03:03.360
Opus 4.8 cost about a $150 GLM 5.2 unfriendly AI cost 27 cents about 5.6 times cheaper.

s29
00:03:03.360 --> 00:03:05.599
So this is the promise I mentioned earlier.

s30
00:03:05.599 --> 00:03:12.159
Open rate models give you frontier quality agents at a fraction of cost.

s31
00:03:12.319 --> 00:03:15.519
But motor cost is only one part of the story.

s32
00:03:15.519 --> 00:03:22.800
To make agent actually fast and reliable, the inference stack itself has to change.

s33
00:03:22.800 --> 00:03:28.000
So let's look at what actually happens inside an agentic workload.

s34
00:03:28.400 --> 00:03:31.040
So first let's look at changes in the workload.

s35
00:03:31.040 --> 00:03:34.480
In the past the dominant usage was chat.

s36
00:03:34.480 --> 00:03:36.640
The basic unit was a request.

s37
00:03:36.640 --> 00:03:42.239
A person asks a question, the model answers and the person reads it.

s38
00:03:42.239 --> 00:03:46.720
Latency meant how fast did I get one response.

s39
00:03:46.720 --> 00:03:48.080
Agents are different.

s40
00:03:48.080 --> 00:03:50.480
The basic unit is a task.

s41
00:03:50.480 --> 00:03:58.239
A task may involve many model cores, many tool cores and it may run autonomously for a while.

s42
00:03:58.239 --> 00:04:04.000
So the user does not really care about the latency of one individual request.

s43
00:04:04.000 --> 00:04:08.799
The user cares about when the whole task is completed.

s44
00:04:08.799 --> 00:04:15.120
That means we have to optimize for tasks, not just individual requests.

s45
00:04:15.840 --> 00:04:18.959
Let's look at agent workflows more closely.

s46
00:04:18.959 --> 00:04:23.360
An agent really runs a session made up of tasks.

s47
00:04:23.360 --> 00:04:25.919
Each task typically runs in a loop.

s48
00:04:25.919 --> 00:04:29.520
First, it plans, which usually means an LLM call.

s49
00:04:29.520 --> 00:04:32.720
Then, it acts maybe by calling a tool.

s50
00:04:32.720 --> 00:04:42.400
Then it observes the result and adds the that back into the context and it repeats this until the task is done.

s51
00:04:42.400 --> 00:04:50.080
So we are constantly alternating between LLM inference and one or more nonLM tool executions.

s52
00:04:50.080 --> 00:04:54.320
So there is a gap between LLM calls.

s53
00:04:54.320 --> 00:05:00.000
An agent can also create sub agents and run them in parallel.

s54
00:05:01.120 --> 00:05:04.479
Agent inputs also look very different from chat.

s55
00:05:04.479 --> 00:05:16.400
The graph here shows the prompt and completion length distributions of our internal coding agent runs with GLM 5.2 which we use day-to-day.

s56
00:05:16.400 --> 00:05:18.000
They are much longer.

s57
00:05:18.000 --> 00:05:25.280
They grow as the task progresses since every observation gets appended back into the context.

s58
00:05:25.360 --> 00:05:27.840
There's an important pattern here.

s59
00:05:27.840 --> 00:05:32.960
Consecutive agent steps usually share a huge prefix.

s60
00:05:32.960 --> 00:05:40.960
If we recomp compute the same prefix every time, we are burning a lot of compute on work we already did.

s61
00:05:40.960 --> 00:05:46.000
So this is one of the biggest opportunities in agentic inference.

s62
00:05:46.639 --> 00:05:49.199
So how token hungry are agents?

s63
00:05:49.199 --> 00:05:54.160
Now let's look at a long horizon task example like deep research.

s64
00:05:54.160 --> 00:06:03.280
We ran explained the spec decoding framework in VLM using code with GLM 5.2 unfriendly AI.

s65
00:06:03.280 --> 00:06:12.800
There are multiple stages and each stage is composed of sub agents which run multiple inferences and tool cores.

s66
00:06:12.800 --> 00:06:23.120
So it might run tens or even hundreds of inference steps sometimes over minutes or hours and the shared context keeps going the whole time.

s67
00:06:23.120 --> 00:06:28.560
For the user what matters is not the latency of a single token or one core.

s68
00:06:28.560 --> 00:06:33.120
What matters is when is my test completed.

s69
00:06:35.039 --> 00:06:39.759
So agent inference is not just chat with more requests.

s70
00:06:39.759 --> 00:06:41.600
It's a different problem.

s71
00:06:41.600 --> 00:06:43.840
The context grows over time.

s72
00:06:43.840 --> 00:06:47.280
Tour work is interled between model cores.

s73
00:06:47.280 --> 00:06:49.520
The number of model cores depends on the input.

s74
00:06:49.520 --> 00:06:55.520
So you can't really plan around a fixed request rate plan.

s75
00:06:55.520 --> 00:07:01.759
And the real metric is end to end test latency, not a single request latency.

s76
00:07:02.960 --> 00:07:05.120
This is where friendly AI comes in.

s77
00:07:05.120 --> 00:07:20.800
We rebuilt the frontier inference cloud specifically for agentic goal close around the challenges I just walked through and we set one goal optimize end to end test latency the task not just the request.

s78
00:07:20.800 --> 00:07:22.160
So how do we do that?

s79
00:07:22.160 --> 00:07:26.160
Let me show you the key engineering behind it.

s80
00:07:27.120 --> 00:07:30.000
Here's the engineering map for how we think about it.

s81
00:07:30.000 --> 00:07:34.400
We built the stack layer by layer around agent workflows.

s82
00:07:34.400 --> 00:07:38.240
There are four big pillars I'm going to cover today.

s83
00:07:38.240 --> 00:07:56.240
Prefix caching key value in short KB cache management, cash aware routing, agent aware optimization and of course underneath we need model layer optimization like sparse attention for long context

s84
00:07:56.240 --> 00:08:02.800
techniques to reduce errors, fast corners, resilience serving and more.

s85
00:08:02.800 --> 00:08:07.199
In this talk, I'm going to focus on the four pillars.

s86
00:08:08.160 --> 00:08:10.319
Let's start with prefix caching.

s87
00:08:10.319 --> 00:08:17.919
Since Asian steps share a large prefix, we compute key value for that prefix once and cache it.

s88
00:08:17.919 --> 00:08:24.160
Then on later steps, we reuse the cache key value and only process the new suffix.

s89
00:08:24.160 --> 00:08:28.160
Reading from cache is much cheaper than recmp computing prefill.

s90
00:08:28.160 --> 00:08:33.919
So this improves time to first token and reduces compute on every step.

s91
00:08:33.919 --> 00:08:40.080
And the longer the task runs in agents, the more valuable this becomes.

s92
00:08:41.279 --> 00:08:48.959
But caching only works if the KV cache actually fits and can move around efficiently.

s93
00:08:48.959 --> 00:08:52.399
So we need strong KV cache management.

s94
00:08:52.399 --> 00:08:59.839
We use frugal memory management to pack more active context onto each GPU memory.

s95
00:08:59.839 --> 00:09:05.120
We use KB contigation to reduce the memory footprint.

s96
00:09:05.120 --> 00:09:11.120
We you we use hierarchical caching across GPU memory, host memory and disks.

s97
00:09:11.120 --> 00:09:14.320
So we can go beyond GPU limits.

s98
00:09:14.320 --> 00:09:17.279
And we also use distributed caching.

s99
00:09:17.279 --> 00:09:24.080
So one prefix can be served across replicas, not just inside one instance.

s100
00:09:26.080 --> 00:09:29.839
At global cluster scale, routing becomes really important.

s101
00:09:29.839 --> 00:09:38.160
A knife load balancer may spread requests evenly across GPU clusters, but it can destroy cache locality.

s102
00:09:38.160 --> 00:09:42.800
A cache aware router at a global scale does something smarter.

s103
00:09:42.800 --> 00:09:51.120
It sends a request to part that already has the right prefix cached turning a cord prefill into one cache ship.

s104
00:09:51.120 --> 00:09:54.560
At the same time, it still has to balance load.

s105
00:09:54.560 --> 00:09:58.160
So one part doesn't become a hot spot.

s106
00:09:58.160 --> 00:10:06.160
In this example, the two requests of task A go to the same part one for cache locality.

s107
00:10:07.839 --> 00:10:11.519
The next piece is agent aware optimization.

s108
00:10:11.519 --> 00:10:16.000
And this is the next fronture of agentic inference.

s109
00:10:16.000 --> 00:10:21.680
Today, most systems schedule each LLM core as if it were independent.

s110
00:10:21.680 --> 00:10:27.920
They don't really understand that this core is part of a longer agent program.

s111
00:10:27.920 --> 00:10:34.000
But if the optimizer knows the agent level context, we can make better decisions.

s112
00:10:34.000 --> 00:10:49.680
For example, preempting the right work, speculatively prefilling context for an unlikely next step or making a better cache eviction decision based on agent level context.

s113
00:10:49.680 --> 00:10:56.720
So the goal is to reduce ant latency not just make one call look fast.

s114
00:10:57.839 --> 00:11:01.680
When we put all of this together, this is the payoff.

s115
00:11:01.680 --> 00:11:08.079
We are using the same model GLM 5.2 with kilo code to create a simple mobile game.

s116
00:11:08.079 --> 00:11:15.920
We ran the same task with model APIs of friendly AI and another well-known inference provider.

s117
00:11:15.920 --> 00:11:26.320
As you can see, friendly completes the same task end to end tox thanks to our Asenticentric cloud design.

s118
00:11:27.200 --> 00:11:29.519
So what does this unlock in practice?

s119
00:11:29.519 --> 00:11:33.600
a stronger production agent stack.

s120
00:11:33.600 --> 00:11:36.320
Take an agent you already like.

s121
00:11:36.320 --> 00:11:44.320
Now plug in openweight frontier models like GLM 5.2, Minimax and Kimi served on friendly AI.

s122
00:11:44.320 --> 00:11:50.480
The motor gives you frontier quality capability and better economics.

s123
00:11:50.480 --> 00:11:57.519
Friendly AI gives you the speed, reliability and endtoend test performance needed in production.

s124
00:11:57.519 --> 00:12:08.240
That combination, quality, speed, reliability, and cost is what makes agents actually useful and economical in production.

s125
00:12:09.360 --> 00:12:16.560
Friendly AI is currently powering teams in production from AI native startups to global enterprises.

s126
00:12:16.560 --> 00:12:20.240
I'd like to highlight a couple here.

s127
00:12:20.240 --> 00:12:27.040
Hilo is a hugely popular Asantic AI coding tool serving millions of users.

s128
00:12:27.839 --> 00:12:35.360
LG is a global enterprise whose businesses range from electronics to healthcare to energy.

s129
00:12:35.360 --> 00:12:39.360
Very different companies, but they all need the same thing.

s130
00:12:39.360 --> 00:12:44.480
Fast, reliable, cost effective agentic inference.

s131
00:12:45.760 --> 00:12:50.160
This testimonial from our client Kilo says it all.

s132
00:12:50.160 --> 00:12:56.800
Over the past year, Kilo Code has tested several inference providers hosting both open and closed models.

s133
00:12:56.800 --> 00:13:11.040
In a split test of GLM5 usage compared against other thirdparty providers and direct usage from the model lab G.A.I., Friendly AI was consistently seven times faster with a significantly

s134
00:13:11.040 --> 00:13:13.200
lower error rate.

s135
00:13:13.200 --> 00:13:19.440
Today, friendly AI is a core component of the killer stack.

s136
00:13:20.320 --> 00:13:23.680
And you can consume this however fits your stack.

s137
00:13:23.680 --> 00:13:26.560
Model API is the fastest way to start.

s138
00:13:26.560 --> 00:13:30.160
Core frontier open weight models through our serless API.

s139
00:13:30.160 --> 00:13:37.360
Dedicated endpoints give you your own isolated deployment with guaranteed SLAs's for production workloads.

s140
00:13:37.360 --> 00:13:44.720
And BYOG bring your own GPU lets you run friendly inference on your own infrastructure.

s141
00:13:44.720 --> 00:13:48.800
Same stack, three ways to deploy.

s142
00:13:49.839 --> 00:13:53.360
To wrap up, there are three things to remember.

s143
00:13:53.360 --> 00:13:59.600
First, frontier open weight models make production agents economically scalable.

s144
00:13:59.600 --> 00:14:03.120
Second, agents are not just chat with more cores.

s145
00:14:03.120 --> 00:14:10.639
Agentic inference requires optimizing end to end task latency with the challenges I mentioned.

s146
00:14:10.639 --> 00:14:16.000
Third, friendly AI is built as an inference cloud for that word.

s147
00:14:16.000 --> 00:14:21.360
Fast, reliable, cost effective agentic inference.

s148
00:14:23.199 --> 00:14:25.120
Thank you for attending my session.

s149
00:14:25.120 --> 00:14:30.959
If you're building agents, give uh frontier openweight motors a try on friendly today.

s150
00:14:30.959 --> 00:14:34.800
You can get started at friendly.ai in minutes.

s151
00:14:34.800 --> 00:14:35.920
And uh thank you.

s152
00:14:35.920 --> 00:14:37.600
I'll be around after the session.

s153
00:14:37.600 --> 00:14:40.320
Um thank you.

s154
00:14:40.330 --> 00:14:42.330
[applause]

s155
00:14:54.740 --> 00:14:56.740
[music]