WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=sOB3HSiG8vo

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/sOB3HSiG8vo.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:12.880 --> 00:00:14.960
Hi everyone, thanks for joining our talk.

s2
00:00:14.960 --> 00:00:17.680
I'm Lou and this is my colleague Chenu.

s3
00:00:17.680 --> 00:00:27.279
So we today we're going to talk about the in uh we both work on the inference team at OpenAI and today we are going to talk about routing IM inference in production

s4
00:00:27.279 --> 00:00:39.120
specifically how our system evolved from routing based on feedback loops driven by engine signals to a more explicit and a predictable policy which is still informed by engine signals.

s5
00:00:39.120 --> 00:00:43.120
However, it's more like the way we use it is different.

s6
00:00:43.120 --> 00:00:57.520
So uh for the agenda today we're going to begin by introducing the inference load balancer what it is what it does and how it has evolved and then Chenu will walk us through the newer control plane and data plane driven architecture

s7
00:00:57.520 --> 00:01:09.200
uh what are the responsibilities of each and followed by a concrete case study of how we reduce the global network overhead and in the end I return to discuss the protection mechanisms

s8
00:01:09.200 --> 00:01:14.560
that help keep the system stable and production level stress.

s9
00:01:14.560 --> 00:01:20.320
So to begin with what is the inference load balancer and where it sit?

s10
00:01:20.320 --> 00:01:25.280
So this is a very high level diagram of the system we are talking about.

s11
00:01:25.280 --> 00:01:29.200
On the left hand side are the front end clusters.

s12
00:01:29.200 --> 00:01:31.200
Those are the GPU cluster.

s13
00:01:31.200 --> 00:01:45.280
Sorry, those are the CPU clusters that act as gateways into our system and they receive user requests then prepare them into the inference request that can be processed by the inference engines

s14
00:01:45.280 --> 00:01:54.320
and on the right hand side are the engine clusters uh which are usually GPU clusters and each hosting multiple inference engines.

s15
00:01:54.320 --> 00:02:02.320
So that's why we they got the name of engine clusters and as you may already heard nowadays GPUs are pretty popular and expensive.

s16
00:02:02.320 --> 00:02:07.600
So um sitting in the middle it is the IRB or inference load balancer.

s17
00:02:07.600 --> 00:02:15.200
It actually runs on the front end clusters but is also a bridge into our inference stack.

s18
00:02:15.200 --> 00:02:20.080
It has two main responsibilities select an engine and proxing the request.

s19
00:02:20.080 --> 00:02:24.400
For this talk, we are going to focus on the engine selection part.

s20
00:02:24.400 --> 00:02:36.000
So in some ways, IRB resembles a very traditional load balancer because a request usually targets a model and a model is backed by multiple engines.

s21
00:02:36.000 --> 00:02:47.360
They may live on different clusters in different regions or even across the continents because that gives us a good resiliency towards localized degradation or cluster failures.

s22
00:02:47.360 --> 00:03:03.519
However, the inference uh stack or the uniqueness of the inference introduces a lot of nuances like uh it has to consider a bunch of signals reported in real time like the well-known time to first token TTF

s23
00:03:03.519 --> 00:03:11.760
time between output tokens also known as token throughput or time between tokens and other hairness and utilization signals.

s24
00:03:11.760 --> 00:03:22.560
Besides there's a important concept of a KV cache which is also well known but for example when the conversation already has a lot of the useful context cached in one engine

s25
00:03:22.560 --> 00:03:33.040
sending the follow-up turns of the same conversation back to the same engine will avoid recomputation improve efficiency and uh reduce latency.

s26
00:03:33.040 --> 00:03:40.879
So the combination of performance reliability locality cach awareness is what makes it such an interesting problem.

s27
00:03:40.879 --> 00:03:43.519
Uh so how we attempted in the problem?

s28
00:03:43.519 --> 00:03:46.000
Let's take a look at the early days.

s29
00:03:46.000 --> 00:03:52.480
And to be honest, early days in in this industry sounds a lot more historic than it really is.

s30
00:03:52.480 --> 00:04:04.879
And the routing process at that time began with a fear of like each request may not be served by all the engines because of uh constraints such as capabilities

s31
00:04:04.879 --> 00:04:09.360
or due uh restrictions due to compute or data residency.

s32
00:04:09.360 --> 00:04:20.400
And among the remaining engines, IRB used a weighted consistent hashing to select the best destination engine for a request or of for certain user.

s33
00:04:20.400 --> 00:04:24.560
Then the important question becomes where are the WS come from.

s34
00:04:24.560 --> 00:04:28.080
So they were generated by a periodic feedback loop.

s35
00:04:28.080 --> 00:04:41.280
The inference engines as mentioned earlier uh reports all kind of the signals we care about and the controller will periodically smooth out those signals and compute a performance score.

s36
00:04:41.280 --> 00:04:46.320
The performance score then will be compared against the fleet average.

s37
00:04:46.320 --> 00:04:51.120
Then the weight will be adjusted basically for each engine.

s38
00:04:51.120 --> 00:05:06.400
is weight goes up if the performance is better or it goes down when the performance is worse than the fleet average and this generated weight will impact the routing and then it's basically a uh control loop

s39
00:05:06.400 --> 00:05:18.240
um conceptually it is very similar to the P controller and no this P controller will not help you care a Linux process but instead it's a classic control theory technique that continuously

s40
00:05:18.240 --> 00:05:31.680
steering the system towards its desired date and we just borrowed this important concept the proportional part of it and uh applied into our uh our load balancer.

s41
00:05:31.680 --> 00:05:34.080
So it has a lot of nice properties.

s42
00:05:34.080 --> 00:05:53.600
For example, it could combine the useful signals we care about into the single routing decision and because of the it adapt to the observed performance as what we mentioned earlier there's a lot of constraints and those constraints might have the some engines basier because they can serve more requests more kind of request than the remaining

s43
00:05:53.600 --> 00:06:03.759
but those basier signal will be fit into the next loop and resulting in the less constraint less constraint request can go to more of those kind of engines.

s44
00:06:03.759 --> 00:06:13.360
So basically they self balanced out and to some extent this just means we don't need to do a lot of manual intervention and it just works.

s45
00:06:13.360 --> 00:06:20.639
However that kind of adaptability comes with big trade-offs because of the same reasons that it combined so many signals.

s46
00:06:20.639 --> 00:06:29.360
It's also very hard to reason about a particular routing decision or like why search engine get a higher weight than we expect.

s47
00:06:29.360 --> 00:06:37.280
And every time we want to fine-tune towards some aspect, it's all almost impossible to not impacting something else.

s48
00:06:37.280 --> 00:06:51.039
And the load is not always very well very evenly distributed because uh sometimes a model is served by engines on different GPU skills and they have different characteristics.

s49
00:06:51.039 --> 00:07:08.960
Then the problem becomes a lot more trickier and the feedback loop sometimes creates bad oscillations because when you shift an engine away some traffic the engine turns a bit cooler and this signal get fit to the controller.

s50
00:07:08.960 --> 00:07:12.800
The controller now thinks that hey this engine can take a lot more traffic.

s51
00:07:12.800 --> 00:07:20.880
then the some traffic going to be shifted back and forth between a few engines and disrupting the KV cache utilization.

s52
00:07:20.880 --> 00:07:30.080
So all those limitations motivated us to rethink about the architecture and uh see if we have new ways to address the problem.

s53
00:07:30.080 --> 00:07:37.199
So I'm going to hand over to Chenu to uh deep dive into the new architecture we tried out.

s54
00:07:38.880 --> 00:07:50.240
Yeah, thank you L. So I'm going to talk about the architecture of the load balancer and how do we reduce the overall overhead with our routing algorithm.

s55
00:07:50.240 --> 00:07:59.599
The load balancer answers one question uh for each request from a CPU cluster which engine should serve it.

s56
00:07:59.599 --> 00:08:06.240
One most naive baseline might be round robin which send requests across engines evenly.

s57
00:08:06.240 --> 00:08:19.759
But if you think a little bit more that doesn't make sense because engines are not homogeneous they can have different hardware and capacity different health and also different distance from CPU cluster

s58
00:08:19.759 --> 00:08:30.720
also run could break cache locality c related requests that could reuse the same engine cache might be sent to different engines.

s59
00:08:30.800 --> 00:08:39.519
A probably better solution might be for each CPU cluster it choose the best engine from its own local view.

s60
00:08:39.519 --> 00:08:41.519
But that's not enough either.

s61
00:08:41.519 --> 00:08:43.519
Think about one extreme case.

s62
00:08:43.519 --> 00:08:53.440
Multiple CPU cluster route traffic to the same engines independently which could overload that engine while leave other engines underutilized.

s63
00:08:53.760 --> 00:09:05.120
So what we need is a globally optimized solution, a control plane that has a global view for all the CPU cluster and GPU engines and could compute a globally optimized

s64
00:09:05.120 --> 00:09:16.399
routing answers and the data plane can make a routing decision quickly based on the answer proved from the control plane.

s65
00:09:18.399 --> 00:09:22.640
Now let's look inside the control plane and data plane.

s66
00:09:22.640 --> 00:09:29.120
In the data plane there is an engine selector which select engine for each request.

s67
00:09:29.120 --> 00:09:37.519
It read the local routing state which includes the candidate engines and the routing weights for each candidate engines.

s68
00:09:37.519 --> 00:09:41.920
Both of them are refreshed asynchronously in the background.

s69
00:09:41.920 --> 00:09:48.880
So we don't need to ask the control plane before we make a routing decision for each request.

s70
00:09:48.880 --> 00:10:01.839
Also the data plane collects realtime engine signal such as number of ready replica engine house etc to surface fast local guard draw in the control plane.

s71
00:10:01.839 --> 00:10:06.720
The data loader combines those live engine signals and never overhead.

s72
00:10:06.720 --> 00:10:21.839
And with offline regressions of capacity, TTFT and TBOT, the optimizer could turn those data into routing weights and the control plane will publish the routing weight for each data plane to pull.

s73
00:10:21.839 --> 00:10:26.560
In this way, no request need to wait on the data plane.

s74
00:10:26.560 --> 00:10:40.560
The control plane continuously compute the next globally optimized routing way snapshot while the data plane make a routing decision based on the latest snapshot already installed locally.

s75
00:10:42.160 --> 00:10:46.399
In summary, there are three important paths through the system.

s76
00:10:46.399 --> 00:10:49.519
The first path is the inference request path.

s77
00:10:49.519 --> 00:11:03.760
The request arrive to the CPU cluster and the data plane inside that CPU cluster will select engine for that request based on the local routing state and forward the request to the selected engines.

s78
00:11:03.760 --> 00:11:07.120
The second pass is the engine signal pass.

s79
00:11:07.120 --> 00:11:20.560
The system continuously collects real-time engine signal such as TTFT, TBOT, number of radio replica and engine house etc. Boost planes need those real-time engine signals.

s80
00:11:20.560 --> 00:11:29.920
The control plane need them to compute a globally optimized routing way while the data plane need them to serve as fast local.

s81
00:11:29.920 --> 00:11:32.800
And the third path is the routing way pass.

s82
00:11:32.800 --> 00:11:41.920
The control plane compute and publish the routing way and the data plane pull the updates to its local cache.

s83
00:11:41.920 --> 00:11:50.079
So only the first pass is synchronous but it's fast and only local inside the data plane of the CPU cluster.

s84
00:11:50.079 --> 00:11:58.560
The other two loops are asynchronous loop and they are to improve future routing decision.

s85
00:11:59.519 --> 00:12:02.480
So that's pretty much of the architecture part.

s86
00:12:02.480 --> 00:12:04.800
But that still leaves one question.

s87
00:12:04.800 --> 00:12:08.720
How do we compute those routing weights?

s88
00:12:10.480 --> 00:12:15.440
But before answer that question, let's answer another question first.

s89
00:12:15.440 --> 00:12:20.240
Why not just send a request to the nearest engine?

s90
00:12:20.480 --> 00:12:27.279
That's because the traffic demand and GPU capacity are not geographically balanced.

s91
00:12:27.279 --> 00:12:36.560
For example, in region one, CPU cluster A send 90 RPS and the nearby engine A can serve 100 RPS.

s92
00:12:36.560 --> 00:12:49.920
So in this case nearest only is fine while in region two CPU cluster B send 120 RPS and the nearby engine B could only serve 100 RPS.

s93
00:12:49.920 --> 00:13:02.800
So in this case if we insist on keeping everything local the extra 20 RPS need to wait on an overloaded engine B. While in region three we are only using 40 RPS

s94
00:13:02.800 --> 00:13:08.399
of an 80 RPS engine C. That still leaves 40 RPS spare.

s95
00:13:08.399 --> 00:13:16.880
So if we send the extra 20 RPS from cluster B to engine C, that will add network distance.

s96
00:13:17.200 --> 00:13:22.880
But it could also avoid a probably much larger engine side waiting time.

s97
00:13:22.880 --> 00:13:27.600
So in this case, a further engine might be faster end to end.

s98
00:13:27.600 --> 00:13:33.040
That's why we need something better than the nearest only routing.

s99
00:13:35.040 --> 00:13:40.000
Now let's open the black box of the optimizer.

s100
00:13:40.000 --> 00:13:44.320
The optimizer accepts four types of input.

s101
00:13:44.320 --> 00:14:04.160
The request from each CPU cluster, the network latency to each engine, the available engine capacity and health and also the TTFT, TBOT latency profiles that tell us how's the engine side latency change as the low increases.

s102
00:14:04.240 --> 00:14:11.120
And with those input the optimizer turn the input to the output routing weights.

s103
00:14:11.120 --> 00:14:19.440
The routing way say for each CPU cluster what fraction of its traffic should go to each GPU engine.

s104
00:14:19.440 --> 00:14:29.120
And the optimization goal is straightforward is to minimize the expected end to end latency across all routed traffic.

s105
00:14:29.120 --> 00:14:37.680
The important part is that the end to end latency includes both the network distance and the engine side latency.

s106
00:14:37.680 --> 00:14:51.120
That means a nearby engine might be attractive when it still has room to serve traffic while a further engine might be better if all the nearby engines are close to full.

s107
00:14:51.120 --> 00:14:56.320
And the optimizer also need to respect several hard constraints.

s108
00:14:56.320 --> 00:14:59.360
First it need to route all the traffic demand.

s109
00:14:59.360 --> 00:15:05.680
Second it need to ensure all the engines stay within the effected capacity.

s110
00:15:05.680 --> 00:15:09.920
Third it need to keep the routing weights non- negative.

s111
00:15:09.920 --> 00:15:24.320
With this the controller control plane get the routing way from the optimizer and publish them and the data plane pull them and use them to make a globally optimized routing decision.

s112
00:15:24.320 --> 00:15:33.199
And that's pretty much of my part and Lou will continue to talk about the protection mechanisms in the system.

s113
00:15:33.279 --> 00:15:34.399
Thanks Chenu.

s114
00:15:34.399 --> 00:15:45.519
So as AI engineers we all kind of know that production in many cases are not behaving in the most ideal case.

s115
00:15:45.519 --> 00:15:55.360
So clusters can fail, GPUs or individual nodes can degrade degree and networking can just get to all kind of mysterious issues.

s116
00:15:55.360 --> 00:16:02.240
So how do we keep our production system uh heresy as much as possible under the heavy load?

s117
00:16:02.240 --> 00:16:05.279
The first thing we have is the penalties.

s118
00:16:05.279 --> 00:16:13.920
Basically when an engine is an outlier, we detect the try to reduce the routing weight to that engine.

s119
00:16:13.920 --> 00:16:28.399
In that way, we give it a chance to either recover by themselves if there's a uh if it's some transient issue or we can have a human intervented out or replace the faulty hardware.

s120
00:16:28.399 --> 00:16:34.480
And secondly, the retries which is a very common technique used to mitigate a problems.

s121
00:16:34.480 --> 00:16:53.199
However, during some cases, it's actually could make them even worse like when the system is very close to like a tip over or very heavy utilized retries, we are send more load and this more load we are cause more failure and cause more retries which is infamous retry storm.

s122
00:16:53.199 --> 00:17:12.720
So we incre implemented caps or budget to constant retries into a acceptable region and this is actually even need to be dynamic because in the happy time or in the normal time we can tolerate a lot more retries than when the system are heavily utilized.

s123
00:17:12.720 --> 00:17:24.400
And finally we have the load shedding which is our last result when the production capac uh capacity couldn't meet the increasing amount of inference demands.

s124
00:17:24.400 --> 00:17:27.919
So we instead try to have all the system fail.

s125
00:17:27.919 --> 00:17:35.520
We basically proactively load shed a portion of the traffic to have the system degraded gracefully.

s126
00:17:35.520 --> 00:17:40.559
So that pretty much concludes our talk today and uh thanks for joining us.

s127
00:17:40.559 --> 00:17:45.360
Uh both of us will be around in our uh booth area this afternoon.

s128
00:17:45.360 --> 00:17:50.559
So if you have further questions, feel free to walk uh to the area and and chat with us.

s129
00:17:50.559 --> 00:17:52.247
Thank you.

s130
00:17:52.247 --> 00:17:54.247
[applause]