WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=7c9FSUVcXR0

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/7c9FSUVcXR0.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:12.559 --> 00:00:14.080
Good morning everyone.

s2
00:00:14.080 --> 00:00:18.400
Welcome to the first infants talk on the last day of AI engineer world fair.

s3
00:00:18.400 --> 00:00:23.039
My name is Nishan Gupta and I'm joined today by my co-speaker Naman Auja.

s4
00:00:23.039 --> 00:00:27.760
We work on building the efficiency training and inference infrastructure at Meta.

s5
00:00:27.760 --> 00:00:32.960
Today we're going to be talking about how do you operate distributed infence systems at scale.

s6
00:00:32.960 --> 00:00:38.000
As we all know, inference is no longer just a research artifact boiled to a product.

s7
00:00:38.000 --> 00:00:43.280
It's a foundational hypers scale infrastructure workload which is growing at a tremendous rate.

s8
00:00:43.280 --> 00:00:53.039
The infence traffic already outpaces the largest microservices in the world and the rate of growth is fastest of any workload we have ever seen.

s9
00:00:54.640 --> 00:01:01.199
Let's rewind back to around 2008 and try to compare the AI era with the cloud era.

s10
00:01:01.199 --> 00:01:05.040
In around 2008, the cloud started as virtual machine offerings.

s11
00:01:05.040 --> 00:01:07.920
The interesting injuring was virtualization.

s12
00:01:07.920 --> 00:01:11.680
Then over time, the value moved up to the stack to the scheduulers.

s13
00:01:11.680 --> 00:01:19.520
Think bog, kubernetes, msos, then to service meshes, then to autoscalers, and then to various platforms that were built on top of it.

s14
00:01:19.520 --> 00:01:24.640
The orchestation layer is what actually captured the value and the complexity.

s15
00:01:24.640 --> 00:01:30.560
AI is on the exact same trajectory but just compressed into a last few years instead of a decade.

s16
00:01:30.560 --> 00:01:33.680
We started with simple models running on GPUs.

s17
00:01:33.680 --> 00:01:45.759
Then we saw the evolution of model serving frameworks like VLM to serve Triton and now we are watching the orchestration layer emerging in real time addressing the complex challenges of routing

s18
00:01:45.759 --> 00:01:51.759
KV cache management prefield decode disagregation and multimodel multiplexing.

s19
00:01:51.759 --> 00:01:57.200
In this next phase of AI era, it's not just about the best models or the kernels or the optimizations.

s20
00:01:57.200 --> 00:01:58.560
It's about the whole ecosystem.

s21
00:01:58.560 --> 00:02:00.719
It's about the control plane and the orchestration.

s22
00:02:00.719 --> 00:02:05.119
And this is what we're going to be focusing in this today's talk.

s23
00:02:06.399 --> 00:02:09.920
So let's talk a little bit about agentic demand explosion.

s24
00:02:09.920 --> 00:02:21.520
In the classical web serving before the AI inference workloads kicked off the capacity scaled roughly linearly with users depending upon the workload type double the users mostly double the QPS

s25
00:02:21.520 --> 00:02:28.319
double the infrastructure fleet if there are no optimizations and capacity planning was mostly a spreadsheet exercise.

s26
00:02:28.319 --> 00:02:38.640
In this new agentic serving capacity is scaling with number of users time number of calls per users times number of tokens which varies depending on the model the optimizations

s27
00:02:38.640 --> 00:02:51.040
the hardware SKU you have a chatbot can have one model call per turn which is evolving now with 10 to 20 for co-pilots and 50 for research agents and now it's

s28
00:02:51.040 --> 00:02:54.879
thousands of such calls for autonomous workflows with no human in the loop.

s29
00:02:54.879 --> 00:03:00.720
The key to take away that you cannot plan capacity for the agents the same way we did for microservices.

s30
00:03:00.720 --> 00:03:07.200
We need to think about elasticity and implement workload aware scheduling and admission control.

s31
00:03:09.120 --> 00:03:19.280
Now let's try to dive a little bit deep into the differences pros and cons differences between the traditional microser in serving versus the modern infence serving across these key dimensions.

s32
00:03:19.280 --> 00:03:33.760
The request shape microservices assumes short uniform request wherever the LLM requests we are seeing in our workloads they can vary from 50 tokens to 100,000 tokens with vastly different compute profiles between prefill and decode stage

s33
00:03:33.760 --> 00:03:48.799
for batching classical stacks for microservices was doing stacking batching mostly at the load balance layer if at all however LLM serving requires continuous inflight batching how otherwise the throughput collapses by an order of magnitude or

s34
00:03:48.799 --> 00:04:03.920
state most of the classical microservices were stateless when we not talking about storage layer however LLM serving does require a huge per request state the KV cache which require which is very expensive to build and even more expensive to throw away

s35
00:04:03.920 --> 00:04:23.919
scaling units when we think about traditional microservices we could run them in cheap CPUs in pods however for model inferencing we require to run on GPUs which are 100 times more expensive which are 10 times slower to acquire and we cannot overprovision them casually otherwise it will lead to a huge wastage.

s36
00:04:23.919 --> 00:04:25.600
failure mode.

s37
00:04:25.600 --> 00:04:34.800
When you think about traditional microservices like most of us have built over the last couple of years, we could even if a micro even if a host crashed or a bot crash, we could restart it.

s38
00:04:34.800 --> 00:04:36.880
We could rebuild the state if needed be.

s39
00:04:36.880 --> 00:04:47.360
However, for the model inferencing, it takes a huge amount of time to go from cold to hot startup and these if and if a GPU is mid decode, it can drop thousands of in-flight

s40
00:04:47.360 --> 00:04:50.479
tokens and it can lead to a queue buildup.

s41
00:04:50.479 --> 00:04:56.320
The takeaway is that the bottleneck is not just the model, it's the orchestration itself.

s42
00:04:58.320 --> 00:05:06.479
Now, as we can see in these hidden decisions behind a prompt, when we go to any gentic application, it requires a bunch of steps which are behind the scenes.

s43
00:05:06.479 --> 00:05:07.440
We have to authenticate.

s44
00:05:07.440 --> 00:05:09.840
We have to choose a model depending on the request type.

s45
00:05:09.840 --> 00:05:12.639
We have to select a region where it goes, we have to do admission control.

s46
00:05:12.639 --> 00:05:14.800
We have to do caching lookup.

s47
00:05:14.800 --> 00:05:17.120
We have to do the we have to run it on the GPU.

s48
00:05:17.120 --> 00:05:18.080
We have to do batching.

s49
00:05:18.080 --> 00:05:19.759
And there's a bunch of other steps involved.

s50
00:05:19.759 --> 00:05:26.000
And as we can see out of all these steps only one step requires the model which is the pre-filled decode.

s51
00:05:26.000 --> 00:05:31.840
If it's disagreed inference or just if it's not disagreed inference other steps require infrastructure.

s52
00:05:31.840 --> 00:05:38.240
The intelligence might lie in the model but the economics the reliability and the user experience are all in the infrastructure.

s53
00:05:38.240 --> 00:05:48.560
And this is why a lot of platform teams across a lot of companies are having much more impact on the product quality and the success ra much more than before.

s54
00:05:50.160 --> 00:05:54.639
Now most of us in this room have a deep expertise across one or two or three layers.

s55
00:05:54.639 --> 00:05:57.199
We might own kernels or kernel optimizations.

s56
00:05:57.199 --> 00:06:03.440
We might own routing or the product itself or we might be operating the GPU infrastructure or the cluster itself.

s57
00:06:03.440 --> 00:06:08.720
But very few of us have operated the whole stack or seen thought about it end to end.

s58
00:06:08.720 --> 00:06:11.600
As you can see in these layers, these layers are not new.

s59
00:06:11.600 --> 00:06:14.080
They have been around for 20 years or more.

s60
00:06:14.080 --> 00:06:18.240
What is new is the combination and the coupling between them.

s61
00:06:18.240 --> 00:06:29.280
A decision at the routing layer can change the cache hit rate at the model layer which can change the batch composition which can change the GP utilization which can change the autoscaling

s62
00:06:29.280 --> 00:06:32.160
decision because of the change in GPUs simplization.

s63
00:06:32.160 --> 00:06:37.199
So everything is entangled when we have any regressions in our inference workloads.

s64
00:06:37.199 --> 00:06:41.120
It's not just about understanding what happened at the caching layer admission control.

s65
00:06:41.120 --> 00:06:43.919
We need to think about the stack top to bottom.

s66
00:06:43.919 --> 00:06:51.919
And when we see any bottleneck, it's very very important to understand at which layer is that bottleneck so that we can invest properly.

s67
00:06:54.160 --> 00:07:05.759
Now diving a little bit deep into a how a prompt works when we have a prompt for any application be it just if you want to generate an image or if you have a research task or if you have complex multi-agent orchestration

s68
00:07:05.759 --> 00:07:08.720
more or less it involves a bunch of these steps.

s69
00:07:08.720 --> 00:07:10.800
The prompt goes to the gateway.

s70
00:07:10.800 --> 00:07:17.120
Then it goes to the router after which it does the cache lookup if the request was already seen before.

s71
00:07:17.120 --> 00:07:33.360
Then it goes to the scheduulers which decides on which GPU cluster it should run on which hardware it can be on Nvidia or AMD or your in-house silicon chip which then goes to the appropriate serving runtime VLM SD lang or whatever we have we are working on and then we stream the response back to the user

s72
00:07:33.360 --> 00:07:41.039
according to the SLO profiles of time to first token and time between each token and while making sure the throughput is what the users desired.

s73
00:07:41.039 --> 00:07:45.599
Now as we can see this inference behaves like a distributed transaction.

s74
00:07:45.599 --> 00:07:47.919
Each arrow in this diagram is a network hop.

s75
00:07:47.919 --> 00:07:50.880
Every of every one of these hops can retry.

s76
00:07:50.880 --> 00:07:52.000
It can time out.

s77
00:07:52.000 --> 00:07:53.120
It can fall back.

s78
00:07:53.120 --> 00:07:54.639
It can even fail.

s79
00:07:54.639 --> 00:08:00.479
And each of these hop will have an SLOs's and it will it it is streaming back to the user.

s80
00:08:00.479 --> 00:08:06.960
So if it fails, the partial fail is mantics are much more harder to deal with than if for a regular RPC call.

s81
00:08:06.960 --> 00:08:16.479
Think about what happens if we have already streamed 200 tokens back to the user and suddenly a GPU host is preempted due to a scheduled or a planned or unplanned maintenance event.

s82
00:08:16.479 --> 00:08:17.599
We cannot just retry.

s83
00:08:17.599 --> 00:08:19.759
We have to think about it holistically.

s84
00:08:19.759 --> 00:08:23.120
This is where this is why reliability we cannot build reliability at the edge.

s85
00:08:23.120 --> 00:08:30.080
It has to be a property of the control plane because the control plane is the one which sees the whole workflow.

s86
00:08:31.840 --> 00:08:37.440
Now let's talk about scheduulers and some of the optimizations uh and how we can think about it.

s87
00:08:37.440 --> 00:08:51.200
So for the traditional microservices we used to think about bin packing of traditional microservices across three four dimensions could be across resource utilization CPU memory or could be across all domains depending upon if you're using AWS if you're using your own in-house

s88
00:08:51.200 --> 00:09:03.040
cloud providers but for inference the schedule needs to be aware AC of across at least seven access when we schedule a particular request it has to be aware of the GPU type there can be any number of heterogeneous

s89
00:09:03.040 --> 00:09:09.279
hardwarees in your cluster H00 versus E00 versus B200 with different network topologies.

s90
00:09:09.279 --> 00:09:20.080
It has to be aware of the HPM headroom, KV cache state, the model weights, whether they are already loaded, whether they're cold or we have to whether they have already warmed up, we have to cold start it.

s91
00:09:20.080 --> 00:09:22.000
It has to be aware of the tenant priority.

s92
00:09:22.000 --> 00:09:27.120
There can be any number of tenants running on that multi-tenant cluster with different SLO profiles.

s93
00:09:27.120 --> 00:09:30.080
We have to also be aware of the workflow context.

s94
00:09:30.080 --> 00:09:39.120
Are we in the third step of reasoning that has already spent X dollar or are we at the initial stages and we can terminate the workflow if it if we are overprovision.

s95
00:09:39.120 --> 00:09:44.320
We have to also think about latency budget depending upon the type of agentic application we are building.

s96
00:09:44.320 --> 00:09:49.920
So this brings us to the idea that we have to make sure that we implement agentic aware scheduling.

s97
00:09:49.920 --> 00:09:59.120
We have to make sure that we place the work which will which will finish the in the fastest and the cheapest time as opposed to just placing it on a random GPU.

s98
00:09:59.120 --> 00:10:09.680
A concrete example might be that the skiller needs to be aware that request R is at step three of a five in a workflow and it has already and step one and two has already spent X plus Y dollar.

s99
00:10:09.680 --> 00:10:14.800
So if a step three fails the whole workflow will be terminated and we have wasted all that compute resources.

s100
00:10:14.800 --> 00:10:24.160
That's why workflow aware orchestration is very very important because it it will change the admission decisions the priority and how we retry.

s101
00:10:25.200 --> 00:10:27.440
Now let's talk about optimizations.

s102
00:10:27.440 --> 00:10:29.839
I will not go too deep into a lot of optimizations.

s103
00:10:29.839 --> 00:10:32.880
There's a lot of research which has already been done outside.

s104
00:10:32.880 --> 00:10:37.600
But I would like to share a framework which at least I like to use when it comes to it.

s105
00:10:37.600 --> 00:10:40.160
And uh we can place it into four quadrants.

s106
00:10:40.160 --> 00:10:42.160
First, can we avoid the work?

s107
00:10:42.160 --> 00:10:48.800
Meaning can we skip it entirely through caching through techniques like prefix caching, response caching, semantic caching.

s108
00:10:48.800 --> 00:10:51.040
The second can we share the work?

s109
00:10:51.040 --> 00:10:54.720
Can multiple requests share compute through batching?

s110
00:10:54.720 --> 00:10:59.680
Think continuous patching, prefill decode, chunk prefill, specular decoding.

s111
00:10:59.680 --> 00:11:01.680
Third, can we move away the work?

s112
00:11:01.680 --> 00:11:07.200
Can we send it somewhere to a cheaper model or closer to the user through mostly routing?

s113
00:11:07.200 --> 00:11:12.079
Can we route it to a smaller model or a cheaper region or some other techniques?

s114
00:11:12.079 --> 00:11:14.000
And lastly, can we delay the work?

s115
00:11:14.000 --> 00:11:25.839
Can we wait for a better moment through admission control and queueing which requires us to understand the priority classes of these requests and implement deadline aware scheduling.

s116
00:11:25.839 --> 00:11:29.839
Now this framework is very powerful because it transfers to various stacks.

s117
00:11:29.839 --> 00:11:35.360
You might be using VLM or SDG lang or tensor RT but every technique fits into one of these quadrants.

s118
00:11:35.360 --> 00:11:45.760
So whenever we think about any optimization to our model we have to apply we have to do a comparison and contrast with the previous techniques and see how all these stack with each other.

s119
00:11:47.200 --> 00:11:51.519
Now whenever we think about scale it's not just important to think about the performance of the model.

s120
00:11:51.519 --> 00:11:54.480
We have to think about the cost economics as well.

s121
00:11:54.480 --> 00:12:02.320
This is where it's important to understand what metric we're trying to optimize because the cost is not just the cost of the GPU or the model.

s122
00:12:02.320 --> 00:12:10.000
It is the cost of all these parameters retries, storage, failures, network and of course the operational cost of development and all that stuff.

s123
00:12:10.000 --> 00:12:17.279
What is important is to understand what is the key key performance indicator for your product which will add value to the users.

s124
00:12:17.279 --> 00:12:22.240
So it's not important to optimize just cost per token or cost per request.

s125
00:12:22.240 --> 00:12:26.320
We have to optimize cost per successful task because this is what actually users care about.

s126
00:12:26.320 --> 00:12:33.120
And if you're able to optimize that, the cost for the overall product decreases and the users are much more happier.

s127
00:12:36.560 --> 00:12:43.040
Now let's talk about reliability a little bit about reliability and uh what it means to prevent cascading failures.

s128
00:12:43.040 --> 00:12:46.639
Now the failure story is never a GPU is preempted or GPU has died.

s129
00:12:46.639 --> 00:12:49.920
The interesting stories is the feedback loop that follows.

s130
00:12:49.920 --> 00:13:03.200
So a GPU can degrade, the latency can rise, the client retries, the Q depth increases, the healthy GPUs will saturate which will follow more retries, much more more full regional failures.

s131
00:13:03.200 --> 00:13:05.600
This is the classic cascading failures.

s132
00:13:05.600 --> 00:13:13.440
But for with a twist for agentic application, the KV cache, we cannot just casually restart or reroute to a different cluster.

s133
00:13:13.440 --> 00:13:20.320
A cold pool has to warm up before it can absorb traffic during which the hot pool has to take on all that all those requests.

s134
00:13:20.320 --> 00:13:34.000
This is why it's important to design the loop breakers very deliberately the circuit breakers at the routing layer the admission control which rejects rather than just queuing the load shedding tied to QEP not just CPU or memory utilization

s135
00:13:34.000 --> 00:13:41.760
and we have to also think about retry budgets because if you don't think about all these things the cost can scale much much more quickly.

s136
00:13:43.279 --> 00:13:48.880
Now we'll hand it over to my co-speaker Naman to talk about the remaining talk.

s137
00:14:56.240 --> 00:14:57.440
Okay, this works I guess.

s138
00:14:57.440 --> 00:15:04.000
Sorry for so once inference reach a production scale it starts looking much more like a distributed system.

s139
00:15:04.000 --> 00:15:06.800
We are not longer just calling a model.

s140
00:15:06.800 --> 00:15:09.839
It's more like a classic distributed uh system problem.

s141
00:15:09.839 --> 00:15:15.040
So in distributed system we talk about cues, scheduuling, autoscaling, fault isolation.

s142
00:15:15.040 --> 00:15:16.720
These are some of the dimensions.

s143
00:15:16.720 --> 00:15:19.040
Inference has all these problems.

s144
00:15:19.040 --> 00:15:20.240
But there are new constraints.

s145
00:15:20.240 --> 00:15:27.279
Now instead of CPU, memory alone, we have CPU, HVM, KV cache and course per successful task.

s146
00:15:27.279 --> 00:15:31.680
So the operating question becomes how the platform know what to do next.

s147
00:15:31.680 --> 00:15:34.240
And that's where observability comes into play.

s148
00:15:34.240 --> 00:15:35.920
It's not just about dashboards.

s149
00:15:35.920 --> 00:15:48.000
It's about uh how to provide input signal to the control loop telemetry fields analysis analysis drive decisions is change of shoulduling and routing and finally we just rinse and repeat

s150
00:15:48.000 --> 00:15:58.800
um I'll just give an overview what are some important metrics first one is time to first token which tells me about uh how much time does it really take to get the first response

s151
00:15:58.800 --> 00:16:10.800
then we have utilization ratio which tell me whether memory or compute is the bottleneck we have success per dollar that tests us whether platform is actually delivering and working efficiently.

s152
00:16:10.800 --> 00:16:18.880
And finally, we have end to end race latency that tells us what's the time being spent across the full request path.

s153
00:16:19.440 --> 00:16:22.880
There's a core trader between latency, cost, and throughput.

s154
00:16:22.880 --> 00:16:24.399
You cannot just get all of them.

s155
00:16:24.399 --> 00:16:26.720
It's pretty analogous to cap theorem.

s156
00:16:26.720 --> 00:16:33.920
If I increase the batch size, um I improve the throughput and cost efficiency, but I may hurt tail latency.

s157
00:16:33.920 --> 00:16:38.800
If I use speculative decoding, I may improve latency, but there are some extra compute.

s158
00:16:38.800 --> 00:16:42.000
Ultimately, you know, we increase the cost per token.

s159
00:16:42.000 --> 00:16:44.399
And finally, I can use a simple model, smaller model.

s160
00:16:44.399 --> 00:16:48.320
I can reduce the latency and cost, but the response will be of low quality.

s161
00:16:48.320 --> 00:16:52.800
Ultimately, I'll do failure analysis, do retries, which brings the cost back up.

s162
00:16:52.800 --> 00:16:56.320
So, every serving decision moves system somewhere in this triangle.

s163
00:16:56.320 --> 00:16:58.639
And our job is to find the perfect setting.

s164
00:16:58.639 --> 00:17:00.480
It's just an optimization problem.

s165
00:17:00.480 --> 00:17:05.280
Now this is where the industry is heading right now.

s166
00:17:05.280 --> 00:17:07.919
Inference needs its own control plane.

s167
00:17:07.919 --> 00:17:14.559
Everything we discussed routing, batching, caching, scheduuling, reliability, they cannot be a separate knob.

s168
00:17:14.559 --> 00:17:17.199
Now they're converging into a logical layer.

s169
00:17:17.199 --> 00:17:19.600
Let's call it inference control plane.

s170
00:17:19.600 --> 00:17:21.919
We used to manage VMs before in distributed system.

s171
00:17:21.919 --> 00:17:27.280
We have autoscaling scheduulers and we had Kubernetes which turned these into control plane.

s172
00:17:27.280 --> 00:17:30.320
Inference is going through the same transition now.

s173
00:17:30.320 --> 00:17:32.240
uh models are becoming resources.

s174
00:17:32.240 --> 00:17:36.480
GPU, KV cache, token, latency, cost are now scheduled around.

s175
00:17:36.480 --> 00:17:42.400
The control plane decides which models serve which request and how it is batched.

s176
00:17:42.400 --> 00:17:50.559
So whether we build this layer internally, use open source or from a vendor, the key design is to assume layer will exist.

s177
00:17:50.559 --> 00:17:57.600
Now let's discuss about where are some of the operating lessons we have experienced in a infra and how they are applicable here.

s178
00:17:57.600 --> 00:18:03.280
The first lesson is that infrastructure bottlenecks usually show up before model bottlenecks in production.

s179
00:18:03.280 --> 00:18:08.400
Many failures can come but there can be just aboutuling and routing or capacity breakdown.

s180
00:18:08.400 --> 00:18:10.320
So these are not related to inference.

s181
00:18:10.320 --> 00:18:12.480
It's about infrastructure problems.

s182
00:18:12.480 --> 00:18:14.240
Then we have elasticity.

s183
00:18:14.240 --> 00:18:15.679
We need elasticity in a system.

s184
00:18:15.679 --> 00:18:19.039
We can have more GPUs but this will not really solve the problem.

s185
00:18:19.039 --> 00:18:21.440
We are just hiding the problem.

s186
00:18:21.440 --> 00:18:26.720
Then our solution to schedule the decision overpowered the efficiency.

s187
00:18:26.720 --> 00:18:32.400
The same fleet can deliver very optimally depending on how you're shoulduling it or how you're batching it.

s188
00:18:32.400 --> 00:18:35.760
Then we have control loops which beats the manual process.

s189
00:18:35.760 --> 00:18:40.640
The platform has to sense, detect and automatically adapt to the system.

s190
00:18:40.640 --> 00:18:44.240
So main takeaway is do not optimize for tokens.

s191
00:18:44.240 --> 00:18:47.440
Optimize for successful task.

s192
00:18:49.280 --> 00:18:51.840
So this is a broader shift I want to leave you with.

s193
00:18:51.840 --> 00:18:54.640
The first phase of a infrastructure was about better model.

s194
00:18:54.640 --> 00:19:00.960
We invest a lot of time in improving our models, smarting, making the model smarter and coming with better benchmarks.

s195
00:19:00.960 --> 00:19:08.000
The current phase right now is faster inference, lower latency, better batching and better GP utilization.

s196
00:19:08.000 --> 00:19:10.640
But the next phase is about orchestration.

s197
00:19:10.640 --> 00:19:13.440
That means GPU, memory, cache and everything.

s198
00:19:13.440 --> 00:19:17.360
These are just resources and they need to be scheduled and controlled.

s199
00:19:17.360 --> 00:19:21.840
The teams that understand this early on will build infrastructure for the future.

s200
00:19:21.840 --> 00:19:26.240
So the closing idea is infrastructure is no longer a server problem.

s201
00:19:26.240 --> 00:19:29.120
It's an orchestration problem.

s202
00:19:29.120 --> 00:19:31.360
Thank you.

s203
00:19:31.808 --> 00:19:33.808
[applause]