WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=cQQbJqvZkpo

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/cQQbJqvZkpo.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:13.120 --> 00:00:14.639
Good afternoon everybody.

s2
00:00:14.639 --> 00:00:20.880
Um I'm Satan Shu from Kov going to be talking about vertical uh mobility.

s3
00:00:20.880 --> 00:00:31.119
It's a quite a fancy topic uh the title that we came up with but basically going to be talking about the inference platform that we have at Corv that we are building to serve

s4
00:00:31.119 --> 00:00:34.559
small to big models and various different types of workloads.

s5
00:00:34.559 --> 00:00:36.960
Uh a quick intro about me.

s6
00:00:36.960 --> 00:00:40.079
Um I joined core just about four months back.

s7
00:00:40.079 --> 00:00:51.200
Uh leading all of inference over there and um before this I was managing everything at AWS Anapuna labs for uh training and before that inference and training at SANOVA.

s8
00:00:51.200 --> 00:00:55.360
So quite a bit of experience in uh in this particular space.

s9
00:00:55.360 --> 00:01:11.360
Um what I will the way I'll be taking you through is explaining to you the consumption models that we have and from that how we have derived what the platform should look like so that we do not need to keep changing the platform and we keep making enhancements in the platform that we have for serving inference

s10
00:01:11.360 --> 00:01:15.680
and how and why the why performance plays such an important role over there.

s11
00:01:15.680 --> 00:01:21.220
Uh I think a little bit of this might be common with the previous topic that was discussed over here.

s12
00:01:21.220 --> 00:01:21.759
[snorts]

s13
00:01:21.759 --> 00:01:24.000
Um consumption models.

s14
00:01:24.000 --> 00:01:28.479
So we have uh at large two biggest consumption models.

s15
00:01:28.479 --> 00:01:36.159
One is a serverless which is where the customers can come in, consumers can come in uh do not need to worry about managing the hardware themselves.

s16
00:01:36.159 --> 00:01:40.640
Do not need to worry about managing the clusters orchestration anything at all.

s17
00:01:40.640 --> 00:01:46.799
There's API, there's UI, you come in, you pay per token uh and you get your model served.

s18
00:01:46.799 --> 00:01:57.520
Um biggest thing over here is that uh the type of models uh that we serve in the catalog that is the breadth of models that the customer will be able to go through.

s19
00:01:57.520 --> 00:02:04.000
Uh I'll talk about dedicated and then I'll come back to serverless because there is one thing unique on the serverless side.

s20
00:02:04.000 --> 00:02:12.480
Dedicated inference service that we provide is more for customers who want to know exactly what hardware they are going to be using and running on.

s21
00:02:12.480 --> 00:02:15.040
Uh but the model deployment also depends on them.

s22
00:02:15.040 --> 00:02:19.040
So they use our service, they use our orchestration layers.

s23
00:02:19.040 --> 00:02:22.160
Uh but the model deployment depends on them.

s24
00:02:22.160 --> 00:02:29.680
Model performance also depends on them as long as we provide in the platform the capability and the knobs to serve those features.

s25
00:02:29.680 --> 00:02:42.160
Coming back to serverless, uh one of the interesting pieces over here is um typically serverless models are very are noisy neighbor problems where uh if let's say everyone is banging on the exact same model

s26
00:02:42.160 --> 00:02:46.800
then you might be timing out quite a bit depending on how much capacity I have behind it.

s27
00:02:46.800 --> 00:02:51.840
So another feature that we have on the serverless side is what we calling provisioned throughput.

s28
00:02:51.840 --> 00:03:01.280
So as a customer, if you know your traffic profile and if you can let us know about that, we can carve it out specifically for you behind the scenes.

s29
00:03:01.280 --> 00:03:08.159
You still do not need to worry about what hardware it is exactly running on as long as your throughput, your SLAs's are maintained.

s30
00:03:08.159 --> 00:03:19.360
So that is another one on the serverless side and that is still charged by by the token but you know that you're not running into the noise enabled problem over there.

s31
00:03:20.159 --> 00:03:34.319
Um let me take a quick stab at few different types of workloads workload shapes that uh we have um that we are seeing and the ratio between these is like continuously changing though agentic is like really high up there.

s32
00:03:34.319 --> 00:03:37.040
Uh, agentic and chat kind of very similar.

s33
00:03:37.040 --> 00:03:42.319
Super high on the input sequence lengths, very low on the output sequence lengths typically.

s34
00:03:42.319 --> 00:03:52.640
But the biggest difference between agentic and chat being the fact that the multi-turns in agentic are super low latency versus in chats because when you get the response as a user,

s35
00:03:52.640 --> 00:03:55.519
you have to read the answer and then you respond to it.

s36
00:03:55.519 --> 00:04:11.599
So there are there are differences over there and that big difference ultimately converts into something uh related to the KV cache management uh but these are these two are both real time and another real-time workload is your voice and videos

s37
00:04:11.599 --> 00:04:22.479
uh which are study streaming and super latency sensitive on the agent agent and chat side um largely the requirements are from throughput point of view not so much from latency

s38
00:04:22.479 --> 00:04:26.880
but real-time voice and videos are absolutely totally latency sensitive.

s39
00:04:26.880 --> 00:04:32.720
Coming to batch, batch is where the SLAs's are like super loose.

s40
00:04:32.720 --> 00:04:35.919
They run into like way like seconds and minutes.

s41
00:04:35.919 --> 00:04:38.000
Sometimes for some customers actually even in hours.

s42
00:04:38.000 --> 00:04:47.360
They're like I'll just throw give me 10 to 12 hours of workload capability and uh I'll throw whatever I can process it whenever you can.

s43
00:04:47.360 --> 00:04:59.360
uh these batch workloads um the way they come into the picture over here in um deciding uh sorry being the requirement for some of the design choices that we make in the stack.

s44
00:04:59.360 --> 00:05:08.880
Imagine these uh four different types of workload shapes um in the time dimension you have to play the game of tetris on how you can fit it in to utilize the underlying infrastructure

s45
00:05:08.880 --> 00:05:11.120
the most.

s46
00:05:12.880 --> 00:05:17.440
I'll give a high level on how our stack is shaped right now.

s47
00:05:17.440 --> 00:05:21.840
Um, and I'll walk you through a bit of a request flow over here.

s48
00:05:21.840 --> 00:05:32.639
So for both serverless and dedicated, if you look at the right hand side of the screen, you'll see that on the platform side we you'll go to the control plane to uh to have your uh authorizations,

s49
00:05:32.639 --> 00:05:46.400
your rate limitings and your usage being tracked etc. so that we can be built accordingly and uh observability so that we can make sure that we are not violating the SLAs's that have been signed

s50
00:05:46.400 --> 00:05:59.440
right uh underlying on the platform I've shown at a super high level that we have these different um inference engines VLMs SG langs and tensorl but there are quite there's quite a few quite a bit of detail over here

s51
00:05:59.440 --> 00:06:09.759
that I'll touch upon uh and underlying that what I'm trying to show over here in green is um various different pieces of hardware.

s52
00:06:09.759 --> 00:06:20.559
So it's not that um so the platform needs to be capable enough of share of having the workload getting distributed across various different generations of these GPUs

s53
00:06:20.559 --> 00:06:20.800
[snorts]

s54
00:06:20.800 --> 00:06:36.960
uh specifically Nvidia GPUs that we use right so let's take up a let's take a few examples uh over here um let's say the request originates from the client side through apps or notebooks any of those or through the agents right it hits the gateway

s55
00:06:36.960 --> 00:06:48.880
once it hits the gateway then uh like I mentioned on the control plane goes through authentication etc etc etc but then um comes either the serverless or dedicated so in the case of serverless

s56
00:06:48.880 --> 00:06:58.960
it'll be paper token so that the token usage would be monitored over here not the exact tokens but just the token usage because we maintain ZDR zero data retention policies

s57
00:06:58.960 --> 00:07:10.240
uh it is multi depending on the multi-tenency or the provisioned uh if it is provisioned then we know underlying for the router it needs to go in and target the

s58
00:07:10.240 --> 00:07:14.160
uh the explicit deployments for the provision throughput uh customers.

s59
00:07:14.160 --> 00:07:17.840
For the multi-end customers, there are separate deployments.

s60
00:07:17.840 --> 00:07:28.880
Router over here specifically uh the the router is very important since um the router is responsible for making KV cache aware routing choices.

s61
00:07:28.880 --> 00:07:30.000
Why is it important?

s62
00:07:30.000 --> 00:07:46.000
Because like I mentioned when we were discussing the workload uh profiles uh the agentic use cases are typically super heavy on the input sequence lengths and bulk of the input sequence length about 80 to 90% depending on which company it is depending on the customers

s63
00:07:46.000 --> 00:07:51.120
80 to 90% of it is the same for various different requests.

s64
00:07:51.120 --> 00:08:04.639
So there is no point in going in and recomputing the prefill or redoing the prefill for that prefill is supercomputebound very expensive that's why as much as you can hit the cache

s65
00:08:04.639 --> 00:08:16.000
more you can save which is why if you look at the token pricing anywhere uh there's a specific price for input tokens and there's a way cheaper price for the cash input tokens so

s66
00:08:16.000 --> 00:08:31.759
caching becomes like really important over here underlying uh The underlying how you want to split the hardware is totally dependent on the choice in the platform and we provide the capability to do either either do a pre-filled decode disagregation

s67
00:08:31.759 --> 00:08:41.680
if the use case desires it or do not do it because prefilled decode disagregation is not uh cheap for every type of use case.

s68
00:08:41.680 --> 00:08:44.800
Uh let's take another request flow over here.

s69
00:08:44.800 --> 00:08:49.200
Let's see if uh when it was a dedicated customer then what will happen?

s70
00:08:49.200 --> 00:08:55.600
uh dedicated customer again will go through the gateways that have been set up for them with proper isolations.

s71
00:08:55.600 --> 00:08:58.399
Um billing is not based on tokens.

s72
00:08:58.399 --> 00:09:02.959
Billing is based on usage of per GPU per hour.

s73
00:09:02.959 --> 00:09:06.720
Uh it's a private gateway so that uh there is noisy enable problem.

s74
00:09:06.720 --> 00:09:08.560
No one else can get in.

s75
00:09:08.560 --> 00:09:24.320
uh same router logic over here so that if there are uh cache heavy if there are requests which are very similar then it's it hits the cache most and depending on the deployment that the customer makes in their on their dedicated GPUs

s76
00:09:24.320 --> 00:09:37.120
uh they can decide if they want to do prefill decode disagregation or not they can decide which which engine to use VLM or S lang or tenslm and given the bulk of capacity that the customer has reserved

s77
00:09:37.120 --> 00:09:47.040
they can decide if they want to have just one deployment uh with the ability to scale through the whole cluster or they want to have multiple different models, multiple different deployments.

s78
00:09:47.040 --> 00:10:00.399
Um one thing that I do want to mention about the router over here um is the fact that heterogeneous capacity across different zones and regions is supported.

s79
00:10:00.399 --> 00:10:04.959
It is a little it's quite a bit of a hard problem to load balance across that.

s80
00:10:04.959 --> 00:10:14.560
So the priority order that we typically take is uh first KV cache locality and then the least loaded fallback.

s81
00:10:14.640 --> 00:10:26.079
Um that's that um another request flow that I want to go over here which um might be a little hard to see from the diagram is I want to take the batch workflow.

s82
00:10:26.079 --> 00:10:41.519
For the batch workflow what we would actually do is the underlying capacity that the customer has let's say the same dedicated inference customer uh during US daytime they're running their real-time workloads and from evening to night they want to run batch workloads

s83
00:10:41.519 --> 00:10:46.480
the same capacity after time can be scheduled to run the batch workloads.

s84
00:10:46.480 --> 00:11:02.079
So we provide the capability in the API to tell when to scale up and when to scale down and as per schedule if we can if they tell us that we have to scale down we will scale down and open it up for batch processing through the night.

s85
00:11:05.120 --> 00:11:14.880
I think I've spoken quite a bit about uh optimizations on the KV cache side but I do want to repeat a little bit because this is one of the most interesting pieces.

s86
00:11:14.880 --> 00:11:24.880
Um it if we can hit on the cache more you can you will avoid the cost of prefill which is the most expensive piece over here.

s87
00:11:24.880 --> 00:11:37.680
Um reusing the cache across multiple different turns in your agentic workloads between turns also there is lot of um similar prefill uh that comes in in the input sequence length.

s88
00:11:37.680 --> 00:11:52.079
Um, think about the chat workloads which is where offloading AV cache also becomes extremely important because with cache with with the chat workloads we have a lot of latency between different between multiple turns that we as users put in

s89
00:11:52.079 --> 00:12:02.640
but uh if we completely evict whatever we had in our particular conversation then the next time we ask a question in that same chat it's going to take a little bit longer.

s90
00:12:02.640 --> 00:12:17.120
Uh so instead of actually completely evicting and redoing the prefill again what the the techniques being used are uh maybe using u we we are using our own but externally we know about LM cache and moon cakes.

s91
00:12:17.120 --> 00:12:30.639
Um what we do is we will offload the KV cache to a high bandwidth storage so that we can store a lot of these prefills uh such that whenever the accompanying request comes for that particular conversation

s92
00:12:30.639 --> 00:12:34.399
it can be loaded in right away into the HPM.

s93
00:12:36.160 --> 00:12:41.600
Um on the performance liver I would I just want to mention a few performance livers.

s94
00:12:41.600 --> 00:12:54.800
We've discussed the PD disagulative decoding are others and how to carefully choose the parallelization degrees and the strategies that is actually very important.

s95
00:12:54.800 --> 00:13:01.920
Uh two of the biggest livers that we've been working with are quantization to NVFP4 and spec.

s96
00:13:01.920 --> 00:13:13.279
uh we do provide capability where if the customer has their data set and they want us to train in speculators for their data sets for better acceptance lens which will ultimately make the output throughput significantly

s97
00:13:13.279 --> 00:13:14.399
higher.

s98
00:13:14.399 --> 00:13:16.240
Uh we do have that as well.

s99
00:13:16.240 --> 00:13:18.079
So but that happens async.

s100
00:13:18.079 --> 00:13:28.720
We uh we get the data async we train the speculators async and then we deploy the speculators into the customer deployments uh if that's what they wanted.

s101
00:13:28.720 --> 00:13:30.959
uh you see three screenshots over here.

s102
00:13:30.959 --> 00:13:38.399
I have posted them from uh the last one month one month's worth of work that uh some of us in my team have done.

s103
00:13:38.399 --> 00:13:48.800
You can see we came quickly on top of the leaderboard on Kimmy 2.6 2.7 and um those are those are from artificial analysis and going back to the session before this

s104
00:13:48.800 --> 00:13:53.600
can we trust that that's why for GLM I have the results from open router.

s105
00:13:53.600 --> 00:14:11.760
So artificial analysis when they run benchmarks they're running very specific workloads open router is actual user traffic uh and you can see uh on the open router side weights and biases so the branding is different but weights and bias is basically k we bought weights and biases about a year back

s106
00:14:11.760 --> 00:14:24.480
uh you can see the speed over here that we have from our deployment is pretty close to what fireworks is providing us fireworks fast right but underlying techniques that we are using is what I want to emphasize the most over here

s107
00:14:24.480 --> 00:14:34.320
uh for for performance optimization that becomes critical because ultimately what you want to serve to the customer what we want to serve to the customer is uh price performance benefit

s108
00:14:36.959 --> 00:14:55.040
quick recap um single platform is what I've been trying to emphasize is what I've tried to show uh two different consumption models serverless and dedicated for customers and within serverless I describe two different consumption models as well pay as you go and provision vision throughput if you care about that

s109
00:14:55.040 --> 00:15:02.000
and ultimately compounding the gains uh through performance optimizations uh in the stack.

s110
00:15:02.000 --> 00:15:02.560
That's all.

s111
00:15:02.560 --> 00:15:04.959
Thank you folks.

s112
00:15:18.975 --> 00:15:20.975
[music]