WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=woIYJYd_etI

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/woIYJYd_etI.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:12.960 --> 00:00:14.000
All right.

s2
00:00:14.000 --> 00:00:19.359
Um, so I'm going to go ahead and get started while people get seated.

s3
00:00:19.359 --> 00:00:21.279
Um, we got a pretty tight turnaround.

s4
00:00:21.279 --> 00:00:24.240
A lot of really wonderful workshops at this event.

s5
00:00:24.240 --> 00:00:27.519
Um, so I want to make sure we have plenty of time.

s6
00:00:27.519 --> 00:00:42.320
Uh so I prepared for 10 or 15 minutes of questions and I think it'd be best especially given this is a workshop if people just interrupted me as I was going raise your hand and I'll call on you and answer your question.

s7
00:00:42.320 --> 00:00:46.239
Um and so just like let's keep this interactive.

s8
00:00:46.239 --> 00:00:48.800
Um I think it's it's better that way.

s9
00:00:48.800 --> 00:00:58.079
Um, and it seems like talked to a couple of people, it seems like we got a lot of folks here who know their who know their inference, uh, know their engines, and so we can get a a sophisticated

s10
00:00:58.079 --> 00:01:00.559
conversation going.

s11
00:01:00.559 --> 00:01:06.960
Um, so the title of this talk is what is an inference engine anyway?

s12
00:01:06.960 --> 00:01:19.520
And the goal is to sort of peel back some of the layers that are behind APIs and uh what is go actually going on inside of the code that runs your inference

s13
00:01:19.520 --> 00:01:21.040
uh when you run it yourself.

s14
00:01:21.040 --> 00:01:30.400
So roughly you know there tokens go in tokens come out um and a lot of the time you get to avoid thinking about what happens in between.

s15
00:01:30.400 --> 00:01:38.960
Um there's this really high quality open source software out there for running your uh running your language or multimodal model inference.

s16
00:01:38.960 --> 00:01:42.000
You've got SGA, VLM, Tensor RTLM.

s17
00:01:42.000 --> 00:01:45.759
Many people write similar things to these in-house.

s18
00:01:45.759 --> 00:01:56.720
Uh they design these these uh engines that like at a high level just taking in some tokens putting out some tokens is not that hard.

s19
00:01:56.720 --> 00:02:04.159
Um you can write something like this in using PyTorch and hugging face transformers in like an hour or two especially if you're good with agents.

s20
00:02:04.159 --> 00:02:12.319
Um but making it uh extremely high performance so that you can produce tokens with good tokconomics is the challenging part.

s21
00:02:12.319 --> 00:02:19.360
Um and that d like sort of drives the design and architecture of these systems.

s22
00:02:19.440 --> 00:02:26.720
Um so just as like a very quick sketch um before we jump back to the high level and talk about why why we're here.

s23
00:02:26.720 --> 00:02:33.920
Um the uh these are generally architected as a set of communicating processes.

s24
00:02:33.920 --> 00:02:40.720
One that does IO does HTTP or RPC serving and responds to the outside world.

s25
00:02:40.720 --> 00:02:47.760
Um and then uh there's a little bit of pre and postprocessing with tokenizers and detokenizers.

s26
00:02:47.760 --> 00:02:56.319
uh and this wraps uh the sort of core of the system where the real work gets done um in sort of two pieces.

s27
00:02:56.319 --> 00:03:04.640
One a sort of scheduler that like defines and schedules work to run on the accelerator, usually a GPU.

s28
00:03:04.640 --> 00:03:08.800
Um and then the actual code that runs on the GPU.

s29
00:03:08.800 --> 00:03:19.777
Um and so this these two components in the bottom right here are where the sort of majority of the performance sensitive work gets done and where the complicated problems are.

s30
00:03:19.777 --> 00:03:20.000
[snorts]

s31
00:03:20.000 --> 00:03:25.599
Um so that's where we'll spend the majority of our time uh discussing.

s32
00:03:27.599 --> 00:03:28.560
Um, all right.

s33
00:03:28.560 --> 00:03:42.080
So, with that stage set, um, I think, uh, I've been to AI engineer since it was the summit and talked about, uh, running, uh, running inference, uh, every time.

s34
00:03:42.080 --> 00:03:45.599
And the answer to this question gets better and better each time.

s35
00:03:45.599 --> 00:03:55.360
Uh, the question of like why inference and why you should care about it, why you should think about being uh, an inference engineer who operates inference engines.

s36
00:03:55.360 --> 00:04:10.959
um our uh fearless leader and uh uh one-man conference army Sean Wang uh tweeted recently uh that everybody in AI infrastructure the boring stuff uh like inference engines not the sexy research stuff

s37
00:04:10.959 --> 00:04:23.600
um is finally getting filthy rich um and so what uh I think he's pointing to here is there's tremendous demand uh for these uh to run inference uh there's demand

s38
00:04:23.600 --> 00:04:27.680
at the labs there's demand within companies who want to own their own stacks.

s39
00:04:27.680 --> 00:04:36.080
There's demand to set up and rethink the infrastructure stack and all of this wraps this core inference workload.

s40
00:04:36.080 --> 00:04:44.960
Um but Sean is also pointing out that a lot of what people have been thinking about up to this point um is sort of the research side, the production of models.

s41
00:04:44.960 --> 00:04:48.000
So um why not, you know, why not think about training?

s42
00:04:48.000 --> 00:04:52.720
Why not go to the training engines uh talk with your precious time?

s43
00:04:52.720 --> 00:05:01.919
I think the fundamental reason why this is such a critical place uh for engineering work to get done and why it's such an like cool place to work as an engineer

s44
00:05:01.919 --> 00:05:07.120
is that in the end training is a cost center and inference is a revenue center.

s45
00:05:07.120 --> 00:05:16.560
Um so when you train models um it turns out um people have not generally been able to make businesses work where they sell model weights to people directly.

s46
00:05:16.560 --> 00:05:22.880
Um and so you can't really turn a training setup into like a proper self-supporting system.

s47
00:05:22.880 --> 00:05:28.479
Uh but uh inference once you have a model um is something that people will pay for.

s48
00:05:28.479 --> 00:05:36.880
People will pay for a system to you know take in their tokens and their context to produce new tokens, produce software for them, whatever.

s49
00:05:36.880 --> 00:05:40.000
Um there's a lot of opportunities to do inference.

s50
00:05:40.000 --> 00:05:50.320
we have set up where there are a small number of essentially model foundaries who produce foundation models that are then deployed and customized by inference engineers.

s51
00:05:50.320 --> 00:05:55.360
Um and then even for training inference has uh snuck its way in.

s52
00:05:55.360 --> 00:06:05.280
A lot of the post-training adaptation of these foundation models requires that you generate samples from the model uh in inference and a lot of the same technologies come to play.

s53
00:06:05.280 --> 00:06:08.160
A lot of the same concerns are present.

s54
00:06:08.160 --> 00:06:17.280
Um, and then finally, sort of more selfishly as an engineer, there's like right now inference is still very new, even though we're years into this new technology.

s55
00:06:17.280 --> 00:06:26.160
Um, and so you end up kind of straddling the whole stack from application architecture uh through linear algebra down to like electrons and hardware.

s56
00:06:26.160 --> 00:06:28.639
So it's a an engineer's playground.

s57
00:06:28.639 --> 00:06:32.080
Um, so we'll see a lot of that as we go.

s58
00:06:32.080 --> 00:06:39.840
Um I'll maybe in the interest of time since we kind of had to get started very quickly I'll skip over this but I've been working on inference for a couple of years mostly

s59
00:06:39.840 --> 00:06:50.080
uh at modal um since coming from the sort of training world before that um and helping a lot of teams sort of deploy and optimize their inference on hundreds or thousands of GPUs on the modal platform.

s60
00:06:50.080 --> 00:06:54.240
Um I've also been making a lot of uh inference applications myself.

s61
00:06:54.240 --> 00:06:56.160
This is a little art piece that I made.

s62
00:06:56.160 --> 00:07:01.919
Um, we uh uh had it in the uh Legion of Honor just this past week.

s63
00:07:01.919 --> 00:07:14.000
Um, this art piece takes what it sees, passes it through a vision language model, one of the Quenes running on uh uh served via the SGA inference engine and running on modal.

s64
00:07:14.000 --> 00:07:17.759
Um, it takes that input and it describes it as though it were an art piece.

s65
00:07:17.759 --> 00:07:21.599
So, it produces one of these little museum placard uh type things.

s66
00:07:21.599 --> 00:07:26.639
And as everybody knows, you're not art until somebody writes one of these little placards to describe what you've done.

s67
00:07:26.639 --> 00:07:29.759
Um, so I pointed it I pointed it at the line on the way in.

s68
00:07:29.759 --> 00:07:31.680
Um, and it came up with a cute little description.

s69
00:07:31.680 --> 00:07:43.759
Anyway, so this is the kind of, you know, uh, sort of in the middle of the set of inference applications between the lowest latency and the uh, highest throughput applications that get deployed.

s70
00:07:43.759 --> 00:07:54.879
Um so uh I'll start by talking in more detail about those workloads like what workloads land on these inference engines and then jump quickly jump move from that quickly into architecture

s71
00:07:54.879 --> 00:08:01.680
how these engines are designed some of the key techniques uh that fit within that architecture to make their uh improve their performance.

s72
00:08:01.680 --> 00:08:14.000
Um and then lastly, a key piece for making these things work in production, observability and uh like debugability of these engines for correctness and for performance.

s73
00:08:14.000 --> 00:08:16.319
All right, so let's start with workloads really quick.

s74
00:08:16.319 --> 00:08:20.639
Um uh I like to break things down into kind of three pieces.

s75
00:08:20.639 --> 00:08:26.960
There's chatbot plus, which is like the original killer app of inference, something like chat gypt or clawed code.

s76
00:08:26.960 --> 00:08:32.959
there's a person waiting on the other side interacting with this inference system and providing additional context.

s77
00:08:32.959 --> 00:08:40.240
Um the plus uh indicates that not only does it talk to a person, it also probably interacts with external systems via tool calls.

s78
00:08:40.240 --> 00:08:42.399
Um then you have background agents.

s79
00:08:42.399 --> 00:08:45.839
They often do similar tasks to something like clawed code.

s80
00:08:45.839 --> 00:08:49.680
Um but they do it in the background, not directly in response to a person.

s81
00:08:49.680 --> 00:08:56.560
So something like Devon, Ramp inspect, uh OpenClaw or Herbies for a non-coding case of a background agent.

s82
00:08:56.560 --> 00:09:04.800
Um so uh this is uh characterized by a similar sort of affordances or workloads but a different performance profile.

s83
00:09:04.800 --> 00:09:07.360
And then lastly data processors.

s84
00:09:07.360 --> 00:09:09.920
Um these generally run relatively small models.

s85
00:09:09.920 --> 00:09:14.560
They take unstructured data like a PDF and they turn it into structured data.

s86
00:09:14.560 --> 00:09:16.880
Um they extract some information from it.

s87
00:09:16.880 --> 00:09:19.600
Um usually the sort of thing you might insert in a database.

s88
00:09:19.600 --> 00:09:30.080
This is something like the uh reductto document processing platform or something like Fathom that takes a a video and extracts the transcript.

s89
00:09:30.560 --> 00:09:36.080
Um okay so under the hood what do these uh workloads look like?

s90
00:09:36.080 --> 00:09:41.200
Um you send in some tokens from a client uh that goes to some server somewhere.

s91
00:09:41.200 --> 00:09:46.000
Um for now we're going to focus on cloud deployment of inference because that's where the majority of it happens right now.

s92
00:09:46.000 --> 00:09:53.040
Obviously in the future a lot of these inference engines are going to end up running on people's uh machines like in a couple of years.

s93
00:09:53.040 --> 00:09:56.000
Um yeah, go ahead.

s94
00:09:58.880 --> 00:10:01.680
The size of the screen

s95
00:10:01.680 --> 00:10:03.200
brightness of the screen.

s96
00:10:03.200 --> 00:10:05.920
Um yeah, so the question was about adjusting the brightness.

s97
00:10:05.920 --> 00:10:07.680
I'm not sure if we can uh fix that.

s98
00:10:07.680 --> 00:10:11.440
That's what I get for making dark mode slides, but yeah.

s99
00:10:11.440 --> 00:10:13.200
Um yeah.

s100
00:10:13.200 --> 00:10:13.440
All right.

s101
00:10:13.440 --> 00:10:20.240
So a client submits um an input uh and then it's processed by our engine running on a server somewhere.

s102
00:10:20.240 --> 00:10:27.839
Um first we process the input tokens in the prefill phase and then we generate the output tokens in the decode phase.

s103
00:10:27.839 --> 00:10:48.000
Um, and so we have uh one uh one prefill uh run uh does a forward pass through the model and then multiple decode runs to produce one or a few tokens at a time and then once we're done return them to the user or generate a tool call and interact with an external system.

s104
00:10:48.640 --> 00:11:06.640
Um so with these work these both the external interface and this sort of internal pre-filled decode split defines these uh the sort of metrics that you care about for a workload that sort of tells you how to deploy your inference engine and and how to measure its success and performance.

s105
00:11:06.640 --> 00:11:11.200
So you have something like queries per second in and queries per second produced.

s106
00:11:11.200 --> 00:11:15.920
Um this one pretty tricky to think about uh under control of your users.

s107
00:11:15.920 --> 00:11:24.959
um uh you probably have a better time thinking of it uh at the engine level thinking of how many queries per second you can handle per replica of your inference engine.

s108
00:11:24.959 --> 00:11:27.920
Um and then you have aggregate demand coming from users.

s109
00:11:27.920 --> 00:11:37.120
Uh then with those queries you want to look at how many tokens there are per query um input and output aka prefill and decode.

s110
00:11:37.120 --> 00:11:43.600
Um so this one is tricky because it's not only under control of users but it's also under the control of a model.

s111
00:11:43.600 --> 00:11:49.360
Uh so the determination of when to finish processing a query is not under your deterministic control.

s112
00:11:49.360 --> 00:11:52.560
It's under the control of the model inside the inference engine.

s113
00:11:52.560 --> 00:11:59.600
Um so this is something not both of those sources of nondeterminism means you need to kind of like measure this benchmark it to get it.

s114
00:11:59.600 --> 00:12:08.399
Um and then uh when we do our processing we uh we produce some intermediate artifacts that we want to cache in the KV cache.

s115
00:12:08.399 --> 00:12:15.279
Uh and then the degree to which that cache is reusable is a key component of the definition of our workload.

s116
00:12:15.279 --> 00:12:25.279
Um this is also mostly under control of users but you can get like depending on the workload there's actually like pretty uh relatively fixed patterns of of prefix reuse.

s117
00:12:25.279 --> 00:12:36.480
Um uh and then finally coming from the application layer um you have some uh latency budget some SLO of like how quickly you want to return tokens.

s118
00:12:36.480 --> 00:12:42.720
That could be how quickly you finish something like the prefill phase and have your first tokens ready to send to users.

s119
00:12:42.720 --> 00:12:48.160
Or it can be how quickly you can generate each set of tokens during your decodes and get them back to users.

s120
00:12:48.160 --> 00:12:53.440
So that's your time to first token and your time per output token or inter token latency.

s121
00:12:53.440 --> 00:12:58.240
Uh so this is going to be the like primary constraint that you operate under.

s122
00:12:58.240 --> 00:12:59.519
Okay.

s123
00:12:59.519 --> 00:13:03.839
Um, and we can map those back onto our application archetypes.

s124
00:13:03.839 --> 00:13:07.680
Uh, like chatbot plus background agent, really high prefix reuse.

s125
00:13:07.680 --> 00:13:11.440
There's some interactivity going on whether that's a person or an agent.

s126
00:13:11.440 --> 00:13:17.839
Um, the chatbot plus generally has relatively short decodes, relatively short outputs of tens to hundreds of tokens.

s127
00:13:17.839 --> 00:13:20.399
Um, and a very tight latency tolerance on that.

s128
00:13:20.399 --> 00:13:27.839
Like a person uh wants to see something in a few hundred milliseconds at the very most, or they get bored, uh, they get angry, they turn away.

s129
00:13:27.839 --> 00:13:31.839
Um, with a background agent, you maybe have a few minutes to produce a PR, right?

s130
00:13:31.839 --> 00:13:37.600
Um, if it's as high quality as an engineer, maybe as like what would be produced by a good software engineer, maybe you have a few hours.

s131
00:13:37.600 --> 00:13:43.440
Um, and so there's not as tight of a of a budget on your milliseconds of network overhead or whatever.

s132
00:13:43.440 --> 00:13:47.440
Um, and then with a data processor, you generally have very low prefix reuse.

s133
00:13:47.440 --> 00:13:51.200
Um, you are seeing different documents each time.

s134
00:13:51.200 --> 00:13:55.360
There's maybe a system prompt, but it's generally small relative to the other inputs.

s135
00:13:55.360 --> 00:13:57.279
Um, and then you have a very short decode.

s136
00:13:57.279 --> 00:14:01.360
you're producing some structured object that's generally short relative to the input.

s137
00:14:01.360 --> 00:14:05.120
Um the structured version is shorter than the unstructured version.

s138
00:14:05.120 --> 00:14:17.440
Um and then the demand here is less on latency generally and more on aggregate throughput because you're generally like ripping through a database backfill type job in a lot of these.

s139
00:14:17.440 --> 00:14:18.480
All right.

s140
00:14:18.480 --> 00:14:36.320
Um and so maybe just want to call out uh like now that we've looked through this you could see that like even though at the application layer this feels like kind of one it's one single workload from the perspective of the user once you start drilling down into what the engine is doing it really feels like there's

s141
00:14:36.320 --> 00:14:38.480
two subworkloads going on.

s142
00:14:38.480 --> 00:14:43.440
there's the like workload to do your prefills to do the processing of input prompts.

s143
00:14:43.440 --> 00:14:47.600
Um, and then there's work to do the decodes to produce the outputs.

s144
00:14:47.600 --> 00:14:54.079
Um, and these uh they sort of operate semi-independently inside the engine.

s145
00:14:54.079 --> 00:15:00.560
Um, and so need like a sort of like different perspective and they follow often like different code paths.

s146
00:15:00.560 --> 00:15:21.040
Um, and they're sort of defined by the fact that the per user in the decode phase, you do you're doing far less arithmetic, far less math per uh per decode per user than you would do in the prefill phase, which is a lot like much higher arithmetic intensity, a lot more uh uh a lot more math per

s147
00:15:21.040 --> 00:15:22.160
request.

s148
00:15:22.160 --> 00:15:34.079
Um, and so that's your fundamental constraint, but also the sort of space in which you operate in configuring an engine and then like adding features or um, adjusting the engine behavior to serve these two things.

s149
00:15:34.720 --> 00:15:42.079
Okay, so I wanted to make sure we're all on the same page about some of the core stuff about what these engines are actually doing before we peel open

s150
00:15:42.079 --> 00:15:44.560
uh, the insides and start looking at them.

s151
00:15:44.560 --> 00:15:47.680
So what is the architecture of an inference engine?

s152
00:15:47.680 --> 00:15:54.079
Um so um this is a little little diagram showing at a high level what an inference engine does.

s153
00:15:54.079 --> 00:16:03.279
So the goal of an inference server is to take an HTTP request or a gRPC request and turn it into an HTTP response to the client.

s154
00:16:03.279 --> 00:16:07.440
Um and so it does that by passing through all these things down here.

s155
00:16:07.440 --> 00:16:14.000
Um so the idea in this diagram um is that if you follow any set of arrows you get the same thing.

s156
00:16:14.000 --> 00:16:25.839
So uh you can sort of peel back a layer of the arrows and look inside and say all right so what what is the the inference server does this high level thing what's the next level down the inference engine is

s157
00:16:25.839 --> 00:16:36.560
the sort of like interesting part of it is the is the part that is not doing the HTTP serving because we aren't in one of those situations where HTTP serving is the hard part is your bottleneck it's not like a proxy

s158
00:16:36.560 --> 00:16:50.959
um it's uh it's a this is a case where you're handling tens hundreds maybe thousands at most of requests and responses per replica and so general purpose hardware just you know kind of eats this for lunch.

s159
00:16:50.959 --> 00:16:54.160
Um so the real interesting stuff is all at the layer below this.

s160
00:16:54.160 --> 00:16:57.759
Um so at the top is your like your server IO for HTTP requests.

s161
00:16:57.759 --> 00:17:01.839
At the bottom is the actual engine and it's like collection of processes.

s162
00:17:01.839 --> 00:17:04.480
Um and you can kind of break this out into three pieces.

s163
00:17:04.480 --> 00:17:12.640
The request comes in expressed using like sort of human friendly um or external representations.

s164
00:17:12.640 --> 00:17:19.039
So you have Unicode strings, you have PNG images, you have MP4 videos or WebRTC streams.

s165
00:17:19.039 --> 00:17:23.439
Um, and these need to be pre-processed before they can get passed to the actual inference.

s166
00:17:23.439 --> 00:17:26.160
So they get turned into collections of tokens.

s167
00:17:26.160 --> 00:17:37.840
um which then get mapped into tensors on the GPU and then we can actually do the inference part of the inference engine which takes you know in the simplest case it takes a tensor

s168
00:17:37.840 --> 00:17:49.919
with K entries in it and gives you a tensor with K plus one entries in it and these are machine learning models that predict the next token they guess what would the next entry in this big tensor look like

s169
00:17:49.919 --> 00:18:02.080
um based off of the data I've seen in training whether that's data from the internet or data from what a helpful agent would do or RL data um whatever that you learn to predict this next tensor entry.

s170
00:18:02.080 --> 00:18:04.080
So that's the core right there.

s171
00:18:04.080 --> 00:18:07.600
Um and that is what is wrapped by the rest of the inference engine.

s172
00:18:07.600 --> 00:18:10.640
Um so then we have post-processing on the side here.

s173
00:18:10.640 --> 00:18:14.559
So post-processing turns that new tensor into a response.

s174
00:18:14.559 --> 00:18:30.400
Uh and that's generally done by this detokenizer that goes back to the sort of like token space first and then uh and then back to our um back to the human and external system friendly representation in the response so it can be returned to the user.

s175
00:18:31.919 --> 00:18:33.919
question.

s176
00:18:34.960 --> 00:18:41.600
Could you explain the difference between the inference server and the inference engine and where does

s177
00:18:41.679 --> 00:18:42.400
Great question.

s178
00:18:42.400 --> 00:18:42.880
Yeah.

s179
00:18:42.880 --> 00:18:49.679
So the question was what is the difference between an inference server and an inference engine and where does something like VLM lie?

s180
00:18:49.679 --> 00:19:00.720
Um so inference server uh means that you've wrapped some like input output around the core operations of like taking requests and turning them into responses.

s181
00:19:00.720 --> 00:19:08.960
Uh so VLM has both of these like VLM operates as a an HTTP or gRPC server.

s182
00:19:08.960 --> 00:19:18.000
Um and then you the engine layer is sort of like part of the VLM or suing SDK as like VLM.

s183
00:19:18.000 --> 00:19:29.840
Um so most deployments are look like are deployed as a server with like a single request single response kind of interface um as depicted here.

s184
00:19:29.840 --> 00:19:32.080
Um yeah, does that answer your question?

s185
00:19:32.080 --> 00:19:32.960
Yeah.

s186
00:19:32.960 --> 00:19:35.760
Um cool.

s187
00:19:35.760 --> 00:19:44.080
Um I said this earlier but some people have come in since like please interrupt raise your hand with questions if I'm like if there's anything confusing or if you'd want to dive deeper.

s188
00:19:44.080 --> 00:19:44.320
Yeah.

s189
00:19:44.320 --> 00:19:46.320
Rom.

s190
00:19:50.640 --> 00:19:50.880
Yeah.

s191
00:19:50.880 --> 00:19:57.039
So the question from Rustom is is this like a stateful or stateless uh workload?

s192
00:19:57.039 --> 00:20:01.283
So the answer is it is stateless

s193
00:20:01.283 --> 00:20:01.600
[snorts]

s194
00:20:01.600 --> 00:20:03.600
if you don't care about performance.

s195
00:20:03.600 --> 00:20:07.679
Um it's stateful once you do care about performance which is pretty much every case.

s196
00:20:07.679 --> 00:20:20.160
So and the primary state that you accumulate over time for requests is the KV cache of like previously computed things for for a previous request attached to some notion of session.

s197
00:20:20.160 --> 00:20:27.760
um and that handling things like sessions actually lives outside of the inference server/inference engine in general.

s198
00:20:27.760 --> 00:20:39.280
So the an individual engine replica can track this KV cache but is up to the wrapping layer which might be something like Nvidia Dynamo or LMD to handle like

s199
00:20:39.280 --> 00:20:40.880
defining what a session means.

s200
00:20:40.880 --> 00:20:44.880
So it's like the clients in that architecture sort of define the the sessions.

s201
00:20:44.880 --> 00:20:47.120
Um and that is where a lot of the trickiness comes in.

s202
00:20:47.120 --> 00:20:54.799
Um, so we won't talk too much about that because it's kind of outside the there's enough to talk about with engines um already, but yeah, uh, come find me afterwards

s203
00:20:54.799 --> 00:20:57.919
um, if you want to talk about that layer.

s204
00:20:58.640 --> 00:21:00.080
Um, right.

s205
00:21:00.080 --> 00:21:05.520
So this diagram, we also talked about it previously, but I just want to return to it now that we've seen a little bit more.

s206
00:21:05.520 --> 00:21:08.640
So this outer layer is our server IO process.

s207
00:21:08.640 --> 00:21:12.799
This um, is this is specifically SGA's architecture.

s208
00:21:12.799 --> 00:21:21.039
There's some slight differences with VLM, but this server IO process communicates with the outside world and then with our pre-processing and post-processing.

s209
00:21:21.039 --> 00:21:29.520
Those things communicate with auler process um which is the one that defines and schedules work onto the GPU, handles things like queuing.

s210
00:21:29.520 --> 00:21:33.200
Um and then the not ready for that yet.

s211
00:21:33.200 --> 00:21:43.360
Um, and then the one or more processes that are sort of closely tied to the GPU are the ones that do the real work of doing the inference.

s212
00:21:43.360 --> 00:21:54.240
Um, so the yeah, so the key things to pull out here is that if you are an old school ML person who's been in this world for a long time, the model forward passes layer is the one that you're familiar with.

s213
00:21:54.240 --> 00:22:00.320
This is where the vast majority of the sort of like PyTorch uh and similar code shows up.

s214
00:22:00.320 --> 00:22:12.240
Um and this is the pl like the GPU or the accelerator in general is the component that costs the most um and is capable of the most work like paflop

s215
00:22:12.240 --> 00:22:14.559
per second scale operations.

s216
00:22:14.559 --> 00:22:21.360
So this is where the like hardcore shave a shave a microcond performance engineering is focused.

s217
00:22:21.360 --> 00:22:31.200
Uh but critically thisuler process out here or it's equivalent in another architecture is your is the bottleneck to that component.

s218
00:22:31.200 --> 00:22:41.840
Um and so this is it's generally operates um it is literally singlethreaded in implementations we'll talk about but is also like semantically it's single threaded.

s219
00:22:41.840 --> 00:22:46.559
It's this one thing that is defining somebody has to define the work that happens on the on the GPU.

s220
00:22:46.559 --> 00:22:53.520
somebody has to manage a bunch of resources there and there's some some fiddly like tricky things to do concurrently.

s221
00:22:53.520 --> 00:23:07.840
And so this becomes this uh like semantically singlethreaded uh like point of control that can potentially despite the fact that it's not doing much it's not doing as many operations

s222
00:23:07.840 --> 00:23:11.280
uh as the work on the GPU it actually becomes your bottleneck.

s223
00:23:11.280 --> 00:23:13.440
It's the sort of gatekeeper to the GPU.

s224
00:23:13.440 --> 00:23:22.240
And so this is the other place where the majority of like performance engineering and care needs to go in the design and operation of an inference engine.

s225
00:23:22.240 --> 00:23:28.480
Um so we'll see more on that as we go but but that's like the high level takeaway about these things.

s226
00:23:28.480 --> 00:23:38.000
So is the solution to that bottleneck problem?

s227
00:23:38.000 --> 00:23:50.000
Uh yeah so the question was is the solution to that bottleneck to you know operate the schedulers in parallel or um uh or to improve some of the external components that consume from it.

s228
00:23:50.000 --> 00:24:07.120
I think the current state of affairs is that this is not reached the like that you can operate these server processes in Python and you just have to be faster on the host side than the GPU side

s229
00:24:07.120 --> 00:24:22.720
and there's not the like complexity increase that you get from trying to manage like locks or coordination on the GPU resources is much that complex lexity increase is much greater than the gain

s230
00:24:22.720 --> 00:24:32.400
um because even if you were able to get it down to a microscond the work wouldn't start happening until the previous work on the GPU had finished and that's

s231
00:24:32.400 --> 00:24:51.440
you know depends on the particular workload but you're talking generally tens of milliseconds to hundreds of milliseconds and so the like um yeah there are general most of the time you can get around it we we'll talk a little bit more about this so maybe ask again once we get to if if that wasn't satisfying.

s232
00:24:51.440 --> 00:24:54.480
Um but yeah, great question.

s233
00:24:54.480 --> 00:24:56.080
More good questions, please.

s234
00:24:56.080 --> 00:24:56.799
Okay.

s235
00:24:56.799 --> 00:25:01.919
Um so then just looking at that was like the architecture diagram that you think of as your like engineering.

s236
00:25:01.919 --> 00:25:05.440
Um let's look at the like life of a request in this architecture.

s237
00:25:05.440 --> 00:25:10.480
So the um the server like receives an input from the outside world.

s238
00:25:10.480 --> 00:25:12.720
It says okay, we've got a request.

s239
00:25:12.720 --> 00:25:24.960
um it sends it to the tokenizer to prepare or pre-process this request that then goes into the scheduler that decides like okay how many resources is this thing going to consume what resources are available

s240
00:25:24.960 --> 00:25:33.360
um and then that runs this core loop of creating um batches of requests to go in the model runners.

s241
00:25:33.360 --> 00:25:39.760
So generally most of the accelerators like benefit from operating on multiple requests in parallel.

s242
00:25:39.760 --> 00:25:44.320
Um so especially GPUs but also TPUs and and many others.

s243
00:25:44.320 --> 00:25:47.279
So you want to collect a bunch of requests together and run them.

s244
00:25:47.279 --> 00:26:01.760
Um if you're a database person you might imagine like I'm g I'm about to run a sequential scan so I might as well grab like three or four queries that are all going to like read this entire table and do all of their um like you know do all their queries at once.

s245
00:26:01.760 --> 00:26:05.919
Um because the hard part is that like sequential scan of loading the entire table.

s246
00:26:05.919 --> 00:26:08.559
Um so there's a similar thing going on here.

s247
00:26:08.559 --> 00:26:18.960
Um uh so this core loop operates over and over again start a batch and outputs the like uh the tokens and the probability distribution over tokens the the logits.

s248
00:26:18.960 --> 00:26:22.559
Um that comes from the the model runner or runners.

s249
00:26:22.559 --> 00:26:26.960
Um so that operates um sort of continually.

s250
00:26:26.960 --> 00:26:39.520
Uh and then as tokens are ready uh they can be passed to the detokenizer to be like turned back into a proper response to the outside world and then sent back to the server.

s251
00:26:41.039 --> 00:26:50.480
So yeah, this is what's interesting about this stuff is that it is simple relative to what's going on inside databases, file systems, uh like web servers.

s252
00:26:50.480 --> 00:26:57.360
Um, partly because it's fairly new, but partly because there's just this core bit at the middle that you just want to operate at maximum speed.

s253
00:26:57.360 --> 00:26:57.679
Yeah.

s254
00:26:57.679 --> 00:26:59.679
Question.

s255
00:27:03.520 --> 00:27:07.520
The question was, is there communication between the tokenizer and the detokenizer?

s256
00:27:07.520 --> 00:27:14.559
Um, to my knowledge, there is no direct communication between the tokenizer and the detokenizer in SG lang.

s257
00:27:14.559 --> 00:27:18.559
Um, in VLM, they actually live inside the same process.

s258
00:27:18.559 --> 00:27:21.679
the um manager, the engine manager.

s259
00:27:21.679 --> 00:27:25.440
And so like it'll be a little harder to determine if there's any communication there.

s260
00:27:25.440 --> 00:27:28.000
Um but in general, they actually don't need to communicate with each other.

s261
00:27:28.000 --> 00:27:35.919
They just need to operate essentially the same um system for mapping from like inputs to uh tokens or tokens to outputs.

s262
00:27:35.919 --> 00:27:36.240
Yeah.

s263
00:27:36.240 --> 00:27:38.080
Did that answer your question?

s264
00:27:38.080 --> 00:27:40.080
Yeah.

s265
00:27:47.919 --> 00:27:50.320
I'm not sure

s266
00:28:06.240 --> 00:28:06.720
Right.

s267
00:28:06.720 --> 00:28:16.080
So the the question was about basically maintaining coherence between the tokenizer and the detokenizer and how would you handle maybe like ambiguous um like ambiguous inputs.

s268
00:28:16.080 --> 00:28:26.640
Um the answer is that yeah they they run the same like underlying li like underlying libraries like the usually like the tokenizers library from hugging face.

s269
00:28:26.640 --> 00:28:35.840
Um and the I would say the tokenization part is the one that is harder than the detokenization part.

s270
00:28:35.840 --> 00:28:42.720
the detoization part is like a little bit more entirely under under the control of the person implementing the the detokenizer.

s271
00:28:42.720 --> 00:28:44.480
Um yeah.

s272
00:28:44.480 --> 00:28:46.480
Um

s273
00:28:46.640 --> 00:28:48.640
software

s274
00:28:50.559 --> 00:28:50.960
yes.

s275
00:28:50.960 --> 00:28:55.840
So the question was does the software allow the tokenizer and detokenizer to always make the same decisions?

s276
00:28:55.840 --> 00:28:57.520
I would say yes.

s277
00:28:57.520 --> 00:29:01.200
I'm trying I'm like trying to make sure that I'm not overpromising here.

s278
00:29:01.200 --> 00:29:05.440
Um but I don't think you need communication between them to ensure that the same decisions are made.

s279
00:29:05.440 --> 00:29:06.480
Yeah.

s280
00:29:06.480 --> 00:29:06.960
Question.

s281
00:29:06.960 --> 00:29:07.919
Sorry, the person.

s282
00:29:07.919 --> 00:29:09.919
Yeah.

s283
00:29:12.799 --> 00:29:14.799
Yes.

s284
00:29:21.520 --> 00:29:21.919
Right.

s285
00:29:21.919 --> 00:29:35.679
So first pointing out that the tokenizer can be um parallelized and generally is like multiple processes um and yet you still see longer time to first token uh if you have longer input sequences.

s286
00:29:35.679 --> 00:29:36.960
So why is that?

s287
00:29:36.960 --> 00:29:45.679
Um so the answer is that processing a longer sequence of tokens takes longer inside of this core loop.

s288
00:29:45.679 --> 00:29:47.760
Um so it's the like primary driver.

s289
00:29:47.760 --> 00:29:53.360
Um if you have a 100,000 tokens that will take let's call it 10 times 10,000 tokens.

s290
00:29:53.360 --> 00:29:58.080
Um once you get to a certain uh size it's basically linear in the input sequence size.

s291
00:29:58.080 --> 00:30:06.960
Um then in a like running inference engine there's also effects from like you break up the input tokens into multiple pieces and run them as separate batches.

s292
00:30:06.960 --> 00:30:19.279
So you can also get additional if there's like queuing and congestion that adds more to the um uh processing of a uh of a long batch um than to a short batch.

s293
00:30:19.279 --> 00:30:22.320
You experience the queueing delay multiple times possibly.

s294
00:30:22.320 --> 00:30:32.640
Um but maybe the one underlying um point to clarify like producing the first token means producing the first output token not the tokenization process.

s295
00:30:32.640 --> 00:30:36.799
Um and then finally not a part of the question but something I wanted to point out.

s296
00:30:36.799 --> 00:30:44.399
It is interesting that you generally have multiple deto tokenizer processes um and one dtokenizer in the sgang architecture at least.

s297
00:30:44.399 --> 00:30:51.919
Um and the answer there is that input tokens are come in at like a much higher rate than output tokens are produced.

s298
00:30:51.919 --> 00:31:00.000
Um and that's related to the fact that prefill can operate at much like higher rates of total tokens per second than decode can in general.

s299
00:31:00.000 --> 00:31:06.399
Um, and so you there isn't as much pressure on the detokenizer component of the system to operate like at the as quickly.

s300
00:31:06.399 --> 00:31:14.000
Um, and it's also in some it's bottlenecking your response to the user, but it's not bottlenecking the core model forward pass resource.

s301
00:31:14.000 --> 00:31:18.320
So it's also like off the hot path a little bit.

s302
00:31:18.320 --> 00:31:20.799
Um, yeah.

s303
00:31:27.520 --> 00:31:27.840
Yeah.

s304
00:31:27.840 --> 00:31:32.799
So the question was how much of the latency comes from this part in the tokenizer versus this part in the core loop.

s305
00:31:32.799 --> 00:31:34.559
This is going to depend on your workload.

s306
00:31:34.559 --> 00:31:50.559
Um I would say there was a nice thing from uh who was that Cruso I think um that pointed out that the existing tokenizers generally don't show up as a bottleneck

s307
00:31:50.559 --> 00:31:57.360
if you have models that are let's call it like 20 billion parameters total or or above.

s308
00:31:57.360 --> 00:32:06.399
um and that and if you but if you are operating a smaller model and you're operating on very large input contexts then it can show up as a as a bottleneck.

s309
00:32:06.399 --> 00:32:08.000
So that's maybe a good rough guide.

s310
00:32:08.000 --> 00:32:14.799
I would say like I actually have not had to think about the tokenizer latency very much.

s311
00:32:14.799 --> 00:32:20.240
It's generally been prefills and substantially decodes that are your the like latency bottleneck.

s312
00:32:20.240 --> 00:32:27.120
And so decodes if you're producing 1,000 output tokens a second, each one is like one millisecond on average.

s313
00:32:27.120 --> 00:32:41.519
Um and the um but up to like 10 milliseconds, 20 milliseconds um or even more if you have a um lower decode throughput like a large that you're running at lower interactivity.

s314
00:32:41.519 --> 00:32:50.559
And so that very the tokenizers are definitely millisecond scale or less and only happens once um per request.

s315
00:32:50.559 --> 00:32:51.360
Yeah.

s316
00:32:51.360 --> 00:32:54.159
Also notable that nobody talks about tokenizer caching.

s317
00:32:54.159 --> 00:32:57.760
People talk about KV caching but not tokenizer caching even though you could do it.

s318
00:32:57.760 --> 00:33:04.159
Um I'm going to keep going just to make sure we get through more stuff but these have been great questions and so if it's still relevant please ask.

s319
00:33:04.159 --> 00:33:08.000
Um another important question to ask wait why are we doing multipprocessing?

s320
00:33:08.000 --> 00:33:15.919
The key reason is they want we want separate threads of control for each subcomponent for our server IO for our tokenization and detokenization for each model runner.

s321
00:33:15.919 --> 00:33:20.640
Um we want them to be able to sort of operate as independently of each other as possible.

s322
00:33:20.640 --> 00:33:31.840
Um they are many of them the tokenizer and the detokenizer are doing relatively computensive work relative to most of what happens maybe in a in a in a web server.

s323
00:33:31.840 --> 00:33:38.480
Um the model runner is um like interfacing with the GPU or accelerator.

s324
00:33:38.480 --> 00:33:46.159
It interacts with that in a way that's kind of similar to the way you would interact with a nick um in like a web server, which is to say you offload work onto it.

s325
00:33:46.159 --> 00:33:51.519
Um and uh so that one you you're giving it its own event loop.

s326
00:33:51.519 --> 00:33:54.960
Maybe in theory that could be part of like a broader event loop.

s327
00:33:54.960 --> 00:34:06.159
Um uh but it's uh certainly like cleaner to have these separate threads of control and optimize their uh you know optimize their latencies and there's not sufficient uh interprocess

s328
00:34:06.159 --> 00:34:09.599
communication to require them to like say live in the same thread.

s329
00:34:09.599 --> 00:34:19.280
Um but why not threads uh for each of these instead of processes um and one of the fundamental reasons is that Python is single threaded per process.

s330
00:34:19.280 --> 00:34:20.800
This has recently started to change.

s331
00:34:20.800 --> 00:34:24.800
there's still a lot of work for compatibility and performance to be done there.

s332
00:34:24.800 --> 00:34:32.240
Um so that uh because Python has a global interpreter lock for all the threads in the same process, you would like frequently block on that.

s333
00:34:32.240 --> 00:34:35.760
Um so it's so that is another reason to sort of split these up.

s334
00:34:35.760 --> 00:34:47.040
Um but that just uh begs the question um uh or I guess begets the question why Python um and the answer is Python has excellent model or GPU support.

s335
00:34:47.040 --> 00:34:50.720
Uh so it's like easy to run code on accelerators with Python.

s336
00:34:50.720 --> 00:34:59.760
So specifically for that model um uh model for pass component um and so wait but why wait why does Python have excellent support here?

s337
00:34:59.760 --> 00:35:08.079
Partly it's that it's a great language for researchers because it's like ease of use and these uh models are still coming out of a sort of researchy data sciency environment.

s338
00:35:08.079 --> 00:35:19.760
But there's a deeper reason which is that Python has really has kept like really deep C and C++ interop as one of its core design principles and that allows for like much easier communication

s339
00:35:19.760 --> 00:35:29.680
uh with these accelerators and uh with like accelerated code in general um than uh like a maybe equally ergonomic language like JavaScript.

s340
00:35:29.680 --> 00:35:32.240
Um but why not like rewrite it in Rust?

s341
00:35:32.240 --> 00:35:38.160
Um the model worker processes again they're just trying to like tell the GPU what to do.

s342
00:35:38.160 --> 00:35:45.839
So you could like the code on the GPU is written in a compiled language and and like carefully optimized like to within an inch of its life.

s343
00:35:45.839 --> 00:35:55.280
Um but the model worker processes are just organizing that and there's a few tricks like CUDA graphs um that like really just pull all the work out of the worker processes

s344
00:35:55.280 --> 00:35:58.720
and into sort of fast stuff running on the GPU.

s345
00:35:58.720 --> 00:36:01.520
Um but there is theuler on the other hand.

s346
00:36:01.520 --> 00:36:04.960
So theuler just needs to stay faster than that work.

s347
00:36:04.960 --> 00:36:11.119
Um, but as GPUs get faster and as the amount of work on the GPU goes down, the pressure on theuler component goes up.

s348
00:36:11.119 --> 00:36:15.599
Um, so this is actually like a pretty good target for a rewrite it and rust type moment.

s349
00:36:15.599 --> 00:36:17.839
Um, because this part is host side.

s350
00:36:17.839 --> 00:36:23.839
It has to manipulate a small number of GPU resources, but it doesn't necessarily have to know stuff about models and pietorrch.

s351
00:36:23.839 --> 00:36:26.480
Um, and so it's like a thinner rewrite.

s352
00:36:26.480 --> 00:36:29.280
It's like specific per engine.

s353
00:36:29.280 --> 00:36:32.800
um and uh uh and host side only.

s354
00:36:32.800 --> 00:36:44.720
So like in the sort of arc of future inference engines, I'd expect it to be a big target um making that faster now that the the sort of iterative development speed is slowing down a bit.

s355
00:36:45.040 --> 00:36:51.359
Okay, so now let's look inside of the like what what is going on inside of these components.

s356
00:36:51.359 --> 00:36:58.000
Um um mostly mostly focused on the um the model forward pass piece.

s357
00:36:58.000 --> 00:37:00.640
So at the very we're going sort of like inside out.

s358
00:37:00.640 --> 00:37:07.119
So at the very bottom there are kernels that run operations for as part of the model architecture in general.

s359
00:37:07.119 --> 00:37:15.599
Um these are mostly deferred by the engine which is to say like the engine doesn't implement all of the kernels for every operation it needs to support.

s360
00:37:15.599 --> 00:37:22.560
Um it has backends that implement the uh these kernels and it consumes them from external libraries of kernels.

s361
00:37:22.560 --> 00:37:28.960
This is one reason why there isn't much differentiation between the engines on sort of raw GPU performance.

s362
00:37:28.960 --> 00:37:32.400
Um, it's mostly about how good are you at getting out of the way of the GPU.

s363
00:37:32.400 --> 00:37:38.960
Um, there are a few special kernels per engine like tree attention for treebased speculators and SG lang.

s364
00:37:38.960 --> 00:37:51.200
Um, and so there's like little bits where an engine will have a specific kernel, but like if it's useful, if it's popular, it ends up in some kernel libraries so it can be used in lots of places and then they consume it from an external library.

s365
00:37:51.200 --> 00:38:02.720
Um then uh the kernels are selected by this sort of model forward pass code that does control flow of and sort of like selection of which kernels to operate.

s366
00:38:02.720 --> 00:38:13.119
Um and this is this is implemented in each engine in general and it's implemented per model like if you want to add support for a new model architecture this needs to be implemented in each engine.

s367
00:38:13.119 --> 00:38:23.680
um there's generally a reference implementation that comes from a library like transformers um but that is insufficiently optimized um the like the setup there.

s368
00:38:23.680 --> 00:38:33.760
So there's um I guess in suang and vlm there is now a path where you can just use that transformers implementation and they'll maybe do a little bit tiny bit of surgery to see what they can automatically

s369
00:38:33.760 --> 00:38:36.640
kernel fuse or automatically optimize or rewrite.

s370
00:38:36.640 --> 00:38:48.880
Um but in general for most things people want to run you're going to go through this hand handwritten as handwritten as software is these days uh path for um like selecting which kernels to run.

s371
00:38:48.880 --> 00:38:57.599
Um then uh wrapping those model forward passes is our batch construction to select the inputs to that model forward pass.

s372
00:38:57.599 --> 00:39:03.119
Um this is kind of one of the primary sources of special sauce per engine.

s373
00:39:03.119 --> 00:39:06.800
you need to set up this um this sort of like queuing setup.

s374
00:39:06.800 --> 00:39:17.200
Um so you might want to be able to for example not operate separate prefill and decode cues but operate one sort of queue where you can mix prefill and decodes together.

s375
00:39:17.200 --> 00:39:19.520
Um that might have some performance implications.

s376
00:39:19.520 --> 00:39:25.599
So you might also want to be able to like dynamically switch whether you're running pure prefills, pure decodes or mixes.

s377
00:39:25.599 --> 00:39:30.880
Um and then you want to do this I think in the ideal operation of an inference server there is no queuing.

s378
00:39:30.880 --> 00:39:35.839
Um, so as somebody who operates these things, I like I treat queuing as a problem.

s379
00:39:35.839 --> 00:39:39.040
Um, but the engine should be able to kind of do it.

s380
00:39:39.040 --> 00:39:47.680
Um, or may maybe it's too extreme to say that queuing is a problem, but um, you don't want this to be a like yeah a growing queue.

s381
00:39:47.680 --> 00:39:50.320
Um, you want it to be flushed as quickly as possible.

s382
00:39:50.320 --> 00:39:55.040
Um, and then lastly wrapping that you have uh detokenization and tokenization.

s383
00:39:55.040 --> 00:39:58.000
Um, this is like mostly boring I'll say.

s384
00:39:58.000 --> 00:40:02.160
um it's like solved in the existing tokenization libraries outside of the engine.

s385
00:40:02.160 --> 00:40:10.400
I think the exception to this being multimodal inputs and outputs um we now have pretty good open weights multimodal input models.

s386
00:40:10.400 --> 00:40:17.440
Um and if you take a look at some traces you'll see that there's like still many operate opportunities for optimization on this path.

s387
00:40:17.440 --> 00:40:29.680
Um, tokenization of an image is just like a harder problem than tokenization of a um of a string of Unicode bytes like conditioned on already having a a tokenizer.

s388
00:40:29.680 --> 00:40:31.680
There's more more work to be done.

s389
00:40:31.680 --> 00:40:38.400
You might need like ffmpeg for instance to like pull out raw frames to go in to something that turns it into a tensor.

s390
00:40:38.400 --> 00:40:42.640
Um, and so yeah, so that is maybe the most interesting part there.

s391
00:40:42.640 --> 00:40:57.359
Um the one way to sort of conceptualize the structure around the model forward codes is that the model forward code is um like going to call into a couple of different types of libraries

s392
00:40:57.359 --> 00:41:01.119
or directly sort of write its own code to run on the GPU.

s393
00:41:01.119 --> 00:41:08.960
Um so a lot of it goes through like PyTorch code which has its own kernel libraries and and can run things on the GPU.

s394
00:41:08.960 --> 00:41:13.680
Um then there are like specific kernel libraries that you would draw from that aren't in the engine.

s395
00:41:13.680 --> 00:41:22.079
And then finally the engine has its own uh set of like kernels that can directly um you know run code on the GPU.

s396
00:41:22.079 --> 00:41:29.839
Um so picking that apart a little bit the um uh the kernels that exist in their backends.

s397
00:41:29.839 --> 00:41:35.520
The core of language modeling architectures these days look something like this.

s398
00:41:35.520 --> 00:41:47.119
Um there's a sequence of layers uh that you take an input, you pass it, do some cross token computation on the sequence, then you do some per token computation to enrich the sequence.

s399
00:41:47.119 --> 00:41:52.880
Um and then you merge that back in with your representation and you iterate this layer after layer.

s400
00:41:52.880 --> 00:42:02.400
Um and so this splits nicely into the two kind of classes of like really critical fast kernels that are available especially via the backends.

s401
00:42:02.400 --> 00:42:07.040
Um so the cross token computation is your attention or linear attention.

s402
00:42:07.040 --> 00:42:10.000
Um there's a bunch of different backends for attention.

s403
00:42:10.000 --> 00:42:14.000
The flash attention kernels from tree and jaw and others.

s404
00:42:14.000 --> 00:42:16.240
Uh flash and fur from Nvidia.

s405
00:42:16.240 --> 00:42:18.480
Uh cutless also from Nvidia.

s406
00:42:18.480 --> 00:42:22.400
Um the Triton uh implementations of these kernels.

s407
00:42:22.400 --> 00:42:25.280
That's OpenAI's kernel authoring library.

s408
00:42:25.280 --> 00:42:34.480
Um, the TRTLM, the engine has also split out its kernels and they're available um, as as backends for attention.

s409
00:42:34.480 --> 00:42:48.079
I think you'll see many of the same names that show up as attention backends also show up as backends for your matrix multiplications inside of your um, your uh, per token computation which is your MLP layer.

s410
00:42:48.079 --> 00:42:51.200
It's where the biggest uh, gems are.

s411
00:42:51.200 --> 00:42:56.480
Uh it's also in some architectures it's this is a dense MLP a classic neural network.

s412
00:42:56.480 --> 00:43:03.599
In other architectures it's these sparse mixture of experts which basically just says don't send every token to the same neural network.

s413
00:43:03.599 --> 00:43:06.960
Per token decide which smaller neural network to send it to.

s414
00:43:06.960 --> 00:43:18.960
Um so the um the dense MLPs there's a there's a nice uh library for this from uh deepseek deep gem that gets used a lot in sglang as kind of like the default.

s415
00:43:18.960 --> 00:43:32.400
Um in VLM there's some quantization focused backends for this the Marlin and bits and bytes backends on the uh for the group gem you can split this out into

s416
00:43:32.400 --> 00:43:39.520
first like there we have this like routing or communication problem of where do tokens go and how do I bring stuff together at the end.

s417
00:43:39.520 --> 00:43:48.319
Um so that can be done with a separate set of kernels like the uh for this all to all communication and routing like the another one from deepseek DP.

s418
00:43:48.319 --> 00:43:55.040
Um and that could be sort of factored out from the like matrix multiplication step.

s419
00:43:55.040 --> 00:44:07.440
Um that had that can have performance penalties and so you can also do that as like one big kernel um a mega kernel uh which is what is done in the deepseek v4 architecture.

s420
00:44:07.839 --> 00:44:21.359
Um okay so these uh with VLM and SGA lang these are things that you can sort of set and control in SG lang it's like kind of pushed up into your face in the form of configuration at start with smart defaults

s421
00:44:21.359 --> 00:44:31.599
VLM is a lot more focused on the like smart defaults side and then you sort of like patch change things more with um environment variables.

s422
00:44:31.599 --> 00:44:35.839
Um, at least last I checked, you know, these things change every like few days.

s423
00:44:35.839 --> 00:44:37.680
Um, but yes,

s424
00:44:37.680 --> 00:44:47.119
questions are like who decides how much?

s425
00:44:58.240 --> 00:44:58.560
Yeah.

s426
00:44:58.560 --> 00:45:04.079
So the question was basically about what's happening with concurrency with really large numbers of requests.

s427
00:45:04.079 --> 00:45:09.760
Um so that goes back so that's uh much higher level than the kernels and backends.

s428
00:45:09.760 --> 00:45:14.319
Um that's happening sort of at our like batch construction and scheduler level.

s429
00:45:14.319 --> 00:45:28.319
So basically the scheduler is going to do what the best that it can to decide like um which requests to run emp generally emphasizing requests that already have KV cache allocated to them um because then you don't need to recomputee

s430
00:45:28.319 --> 00:45:39.920
um the if you need to evict something from the cache to make space for another request um yeah that will generally get like put out to CPU memory and then disk

s431
00:45:39.920 --> 00:45:49.040
um and I would say Yeah, to my point earlier about queuing, I think of that as essentially like a failure state of the engine.

s432
00:45:49.040 --> 00:45:51.920
Um, that we like I've deployed my engine improperly.

s433
00:45:51.920 --> 00:46:03.839
If I get to the case where I have KV cache thrashing or like things being flushed to disk in the same way that there's like swap memory in your operating system, like you can swap memory to disk, but at that point your MacBook becomes unusable,

s434
00:46:03.839 --> 00:46:04.240
right?

s435
00:46:04.240 --> 00:46:12.640
Um, so it's it's a useful feature to have, but almost like um so that you don't go hard down when you have a serving problem.

s436
00:46:12.640 --> 00:46:25.920
Um yeah, how is mobile?

s437
00:46:29.040 --> 00:46:29.280
Yeah.

s438
00:46:29.280 --> 00:46:32.079
So the question was about like serverless deployments and cold starts.

s439
00:46:32.079 --> 00:46:35.920
I'm not going to talk about that this much because I promised Swix I wouldn't do a vendor talk.

s440
00:46:35.920 --> 00:46:38.160
Um but please talk to me afterwards.

s441
00:46:38.160 --> 00:46:43.040
There's a blog post called truly serverless GPUs that talks about how we solve that problem.

s442
00:46:43.040 --> 00:46:48.640
Memory snapshotting, GPU memory snapshotting, custom file system, bunch of pieces to make the cold starts faster.

s443
00:46:48.640 --> 00:46:50.480
Um, yes.

s444
00:46:50.480 --> 00:46:52.640
Um, cool.

s445
00:46:52.640 --> 00:46:53.200
All right.

s446
00:46:53.200 --> 00:47:03.599
So, let's talk about some of the key techniques that you need uh to implement to in inside an inference engine to make sure like uh your performance is good.

s447
00:47:03.599 --> 00:47:12.319
Um so attention is fundamentally quadratic at least like the uh loadbearing part of the attention in in every architecture that's popular.

s448
00:47:12.319 --> 00:47:26.319
Um and the um so the solution to that is to store things um after you've computed them and you do like a little space-time trade-off for linear computational complexity in exchange for linear storage.

s449
00:47:26.319 --> 00:47:36.880
um that runs into problems with how much can you store in GPU memory because it's actually often slower to store this somewhere else uh and then load it back in as opposed to recomputing it.

s450
00:47:36.880 --> 00:47:38.319
GPUs are fast.

s451
00:47:38.319 --> 00:47:42.400
Um so you want to be efficient with your use of GPU memory.

s452
00:47:42.400 --> 00:47:46.640
Um and then you often have requests that share prefixes.

s453
00:47:46.640 --> 00:47:51.520
So for example like all these commandments start with thou shalt not.

s454
00:47:51.520 --> 00:47:55.839
Um, and so you can reuse the computation of everywhere up to here.

s455
00:47:55.839 --> 00:48:00.560
Um, as soon as a single token differs, you no longer can share the prefix.

s456
00:48:00.560 --> 00:48:03.520
So it's kind of this like tree or try shaped structure.

s457
00:48:03.520 --> 00:48:11.359
Um, but you want to be able to make use of that, especially if you have like a giant system prompt for like an agent system that's tens of thousands of tokens.

s458
00:48:11.359 --> 00:48:14.880
Um, it's like quite nice to be able to share that across users.

s459
00:48:14.880 --> 00:48:21.920
Um and so the solution to this is like KV caching with pages or or radices.

s460
00:48:21.920 --> 00:48:34.559
Uh so you store the information in this um in like a a page cache data structure like you'd have in an operating system and then you just need to reconstruct these tensors sort of on the fly to pass them to attention kernels.

s461
00:48:34.559 --> 00:48:38.720
So that that job is now it used to be the job of the engine to solve that problem.

s462
00:48:38.720 --> 00:48:50.640
That's now incorporated into the attention kernels like flash attention 4. something that our team worked on the um page size one um and sort of like RAD XC form of KV caching you can see on the right

s463
00:48:50.640 --> 00:49:07.040
um and uh yeah so this is this has moved a little bit I put this in here because it used to be like a defining feature of the engine but now it's kind of actually part of the kernels and really it's the management of KV cache capacity and um and the like layout of the KB cache that's

s464
00:49:07.040 --> 00:49:15.839
the problem of the engine um another problem I already talked about how this the scheduler can be this um host side bottleneck.

s465
00:49:15.839 --> 00:49:29.280
What you want when you're running stuff on the GPU like both at the sort of level and then inside of models is that you want that just like at every moment some kernel is running on the GPU and things happen on the CPU to decide what that

s466
00:49:29.280 --> 00:49:30.640
uh should run on the GPU.

s467
00:49:30.640 --> 00:49:33.440
But they shouldn't ever block uh progress.

s468
00:49:33.440 --> 00:49:36.800
But it's very easy to accidentally write some kind of host device sync.

s469
00:49:36.800 --> 00:49:38.880
There does have to be information passed between them.

s470
00:49:38.880 --> 00:49:42.480
the GPU can't just like send the response directly to the user.

s471
00:49:42.480 --> 00:49:45.760
Um, it wouldn't be that good at it if it did run an HTTP server.

s472
00:49:45.760 --> 00:49:48.400
Um, and so you you do need this CPU work.

s473
00:49:48.400 --> 00:49:50.640
You just want to make sure it doesn't block.

s474
00:49:50.640 --> 00:49:54.400
Um, one of the key techniques for this is CUDA capture.

s475
00:49:54.400 --> 00:50:04.880
So you take all of these kernels that get run in the forward pass of the model and you track all of them and you turn that into one big data structure a DAG of operations

s476
00:50:04.880 --> 00:50:10.319
of like this kernel runs and and like you know mutates the data at this pointer.

s477
00:50:10.319 --> 00:50:17.440
Once that is done this kernel can now run to mutate what was present at that same pointer and mix it with data from another pointer.

s478
00:50:17.440 --> 00:50:22.160
So this big DAG of C of of uh operations is your CUDA graph.

s479
00:50:22.160 --> 00:50:34.079
And this um allows you to go from launching like each one of these GPU kernels having a little bit of CPU work to decide what to run, how to run it, uh what you know which pointer to put

s480
00:50:34.079 --> 00:50:43.920
um you turn it into just one CPU side work to decide which graph to launch and then all of it gets launched from that single graph which is what's depicted in this um

s481
00:50:43.920 --> 00:50:45.359
uh peretto trace.

s482
00:50:45.359 --> 00:50:53.839
Uh so this allows you to relax quite a bit about what is going on in the model uh forward pass component at least.

s483
00:50:55.359 --> 00:51:06.240
Um another problem decode here is sequential um and every time you run it you have to load like either all of the parameters of the model or all the active parameters in ane

s484
00:51:06.240 --> 00:51:11.760
architecture um and you need to do that over and over again um by default for each token.

s485
00:51:11.760 --> 00:51:21.599
So you're like on behalf of your users, you might be loading like a terabyte of data or 100 gigabytes of data per token and you want to produce a token every couple milliseconds.

s486
00:51:21.599 --> 00:51:27.760
Um so now you're looking at that's pabyte per second scale uh memory bandwidth.

s487
00:51:27.760 --> 00:51:29.760
That's uh pretty hard to achieve.

s488
00:51:29.760 --> 00:51:36.880
Um so one solution is key technique for solving this is speculative decoding where you take um a separate model.

s489
00:51:36.880 --> 00:51:42.400
The speculator guess what the next couple of tokens will be and then pass it through the target model.

s490
00:51:42.400 --> 00:51:46.160
Um and then the target model sort of grades all of those in parallel.

s491
00:51:46.160 --> 00:51:59.200
Um and if you apply the right sampling techniques, rejection sampling, um then you can uh guarantee that uh the output from the target model is the same as you would have gotten if you'd run it sequentially

s492
00:51:59.200 --> 00:52:04.400
up to numeric um the same output uh but you've now run it in parallel.

s493
00:52:04.400 --> 00:52:08.319
You sort of turn your decodes into tiny prefills which is nice.

s494
00:52:08.319 --> 00:52:13.599
Um and so the um I'll skip over a little any details about the algorithm.

s495
00:52:13.599 --> 00:52:18.160
There's a great we wrote a blog post about this uh just this past week or two.

s496
00:52:18.160 --> 00:52:26.880
The key reason why this is really important is that as you improve the quality of the speculator, as you improve the number of tokens it can guess in a row,

s497
00:52:26.880 --> 00:52:31.040
um you actually get basically a linear speed up in your decode throughput.

s498
00:52:31.040 --> 00:52:45.119
So like many other optimizations that you can do on your inference engine are like the it's like a game of inches of like a percent 5% 10% like each time and you have to do the like very like difficult performance engineering

s499
00:52:45.119 --> 00:52:47.680
for each of those you know percentage points.

s500
00:52:47.680 --> 00:53:06.079
Um but then on the other hand you could sit down and train a better speculator model um and to first approximation throw data and compute at it um and then get a 2x speed up or a 4x speed up or an 8x speed up um on top of the the the baseline.

s501
00:53:06.079 --> 00:53:18.240
Um so this is has emerged as one of the like key techniques to allow you to operate say like a thousand tokens per second on um on Nvidia hardware without have for actually fairly large models

s502
00:53:18.240 --> 00:53:25.839
uh without having to go to um like a a novel accelerator like an LPU or Cabris.

s503
00:53:26.559 --> 00:53:32.800
Um I'm actually going to there's a lot of parallelism techniques but in the interest of time I'm going to kind of skip over this one.

s504
00:53:32.800 --> 00:53:35.920
um you want to be able to split work onto accelerators.

s505
00:53:35.920 --> 00:53:38.960
Um a lot of this comes from the underlying kernels.

s506
00:53:38.960 --> 00:53:41.440
You need good kernels that can operate with this parallelism.

s507
00:53:41.440 --> 00:53:50.559
But within the engine in the sort of model forward passcode which is written on a sort of per engine basis um you need to do all of the communications

s508
00:53:50.559 --> 00:53:51.599
around that.

s509
00:53:51.599 --> 00:53:59.520
Um and so the communications is where a lot of the problems come in in these uh parallelism uh techniques.

s510
00:53:59.839 --> 00:54:00.800
Um all right.

s511
00:54:00.800 --> 00:54:04.559
So, uh, observability with our last couple minutes here.

s512
00:54:04.559 --> 00:54:09.200
Um, inference services like all services will have some bugs in them.

s513
00:54:09.200 --> 00:54:13.119
Um, you have everything from application level bugs from models misbehaving.

s514
00:54:13.119 --> 00:54:16.079
Um, uh, which is really not the engine's problem.

s515
00:54:16.079 --> 00:54:20.480
It's the app developer's problem, but they'll show up on your dashboard sometimes.

s516
00:54:20.480 --> 00:54:22.160
Uh, model quality bugs.

s517
00:54:22.160 --> 00:54:23.440
These are the problem of the engine.

s518
00:54:23.440 --> 00:54:25.119
Are you actually doing your inference correctly?

s519
00:54:25.119 --> 00:54:29.839
When you applied this performance optimization, did you actually guarantee it didn't change model behavior?

s520
00:54:29.839 --> 00:54:32.720
Um, one of the most common ones here is actually on model release.

s521
00:54:32.720 --> 00:54:34.640
Tokenizers are often slightly bugged.

s522
00:54:34.640 --> 00:54:40.079
They both tokenize and apply chat templates and there's often some like uh some issues there.

s523
00:54:40.079 --> 00:54:45.920
Um, they're beaten down in a couple weeks after model release, but it's something you always have to be like on guard for.

s524
00:54:45.920 --> 00:54:49.680
Um, and then performance bugs or engine performance issues.

s525
00:54:49.680 --> 00:54:52.000
The kind of the worst ones are regressions over time.

s526
00:54:52.000 --> 00:55:00.079
as you operate a replica for an extended period, you might see performance degradation from issues that only arise once the thing has been running for an extended period.

s527
00:55:00.079 --> 00:55:10.240
Um then you also might observe, especially in a heterogeneous cloud deployment environment, differences across replicas in their performance um that then need to be investigated.

s528
00:55:10.240 --> 00:55:15.920
So the goal with these as with all kinds of like production debugging is observability.

s529
00:55:15.920 --> 00:55:23.599
Um if you can log enough information uh to debug just from those logs, then you don't have to spin up a development server and take a bunch of extra time.

s530
00:55:23.599 --> 00:55:31.680
Um and so this runs from application level stuff um to I think focusing on the engine problems, model quality.

s531
00:55:31.680 --> 00:55:36.800
You'll you want to be able you want to be good at running evals for model quality as part of your deployment.

s532
00:55:36.800 --> 00:55:43.119
Um so you want to hit your actual deployment with like evaluation benchmarking type scripts.

s533
00:55:43.119 --> 00:55:47.599
Um there's scripts in SUAN, there scripts in VLM, there's external benchmarking tools.

s534
00:55:47.599 --> 00:55:55.119
Um take those uh calculate some numbers, log traces somewhere, log as much data as you can about those so you can debug later.

s535
00:55:55.119 --> 00:56:03.440
Um and then in prod log traces and try to get feedback from users on correctness that you can thread through all the way back to those requests.

s536
00:56:03.440 --> 00:56:06.880
Um for tokenizer bugs, hot tip, log the token IDs.

s537
00:56:06.880 --> 00:56:09.680
Set your engine up so that the token IDs are logged as well.

s538
00:56:09.680 --> 00:56:12.720
Um and then finally for performance, performance bugs can show up anywhere.

s539
00:56:12.720 --> 00:56:24.720
So the more metrics that you log uh the better like more metrics than you think you need um because you never know which crossorrelation will re will make the bug super easy to discover instead of super hard.

s540
00:56:24.720 --> 00:56:28.880
Um so rather than going through those I'm going to talk a little bit about this.

s541
00:56:28.880 --> 00:56:33.040
This is an endpoint dashboard on modal and what like how you might read it.

s542
00:56:33.040 --> 00:56:45.440
Um I was operating that um the uh art project uh that I showed at the beginning and I just actually spun up like three different replicas of it uh to see what would happen like under increased load.

s543
00:56:45.440 --> 00:56:56.799
Um so there was a traffic spike which you can see down here that turns into increasing uh first time to first token as things start to cue uh on the prefill side and and don't enter prefill

s544
00:56:56.799 --> 00:56:59.839
um until uh the things have finished.

s545
00:56:59.839 --> 00:57:12.480
uh and then also shows up as well as inter token latency as uh decodes get first get a little bit slower and then a lot slower with this queuing as there's sort of contention on the underlying GPU.

s546
00:57:12.480 --> 00:57:20.400
Um and so the solution in this case with going back to the question about serverless deployments is this is detected as like increased load.

s547
00:57:20.400 --> 00:57:21.839
You spin up new replicas.

s548
00:57:21.839 --> 00:57:23.680
Those replicas come online.

s549
00:57:23.680 --> 00:57:35.839
They start taking on some of this request load um and then the um like the amount cued um uh decreases back down to baseline and things are um you your congestion is solved.

s550
00:57:35.839 --> 00:57:39.520
So I think um that's the like primary way.

s551
00:57:39.520 --> 00:57:45.520
I think in many cases, you know, we scale up additional resources to solve congestion and queuing.

s552
00:57:45.520 --> 00:57:47.760
Um, so yeah.

s553
00:57:47.760 --> 00:57:50.880
Um, when you need to go deeper, I'll skip over this.

s554
00:57:50.880 --> 00:57:55.920
Um, but there's the NSI systems or torch profiler is one of your key tools here.

s555
00:57:55.920 --> 00:57:58.240
It allows you to see the whole engine at once.

s556
00:57:58.240 --> 00:58:02.960
Um, allows you to see all the processes that are running, all the work that's going on on the GPU.

s557
00:58:02.960 --> 00:58:08.559
Um, and pick out issues like, oh, this GPU is running uh like everything's slower than the others.

s558
00:58:08.559 --> 00:58:11.760
turned out to be a NUMA awareness problem.

s559
00:58:11.760 --> 00:58:13.920
Um, okay.

s560
00:58:13.920 --> 00:58:21.040
So, uh, closing out here, um, if you want to learn more, mini SGA and Nano VLM are a really great place to start.

s561
00:58:21.040 --> 00:58:25.599
These are these simple implementations of the engine architecture, um, by the two teams.

s562
00:58:25.599 --> 00:58:26.799
They're great for humans.

s563
00:58:26.799 --> 00:58:28.480
Um, they're incredible for agents.

s564
00:58:28.480 --> 00:58:32.160
They can fit all the context in, you know, in their context window.

s565
00:58:32.160 --> 00:58:35.839
Uh, Alexa Gordic did a great walkthrough blog of VLM.

s566
00:58:35.839 --> 00:58:38.559
uh and uh share these slides later.

s567
00:58:38.559 --> 00:58:46.559
Um also work through that in a notebook so you can play with it and then actually uh you can point your coding agent at the coding base and ask questions.

s568
00:58:46.559 --> 00:58:58.960
Cognition al also operates this thing called deep wiki which basically crawls these a bunch of repos including sing and vlm and writes nice architecture diagrams and makes this like wiki style

s569
00:58:58.960 --> 00:59:08.559
um page where you can sort of read and ask and then ask their um agent questions which is a nice thing to pair with your own agent setup.

s570
00:59:08.559 --> 00:59:10.480
Um so that's everything.

s571
00:59:10.480 --> 00:59:11.280
I'm out of time.

s572
00:59:11.280 --> 00:59:21.119
So I'll just quickly say if you're interested in deploying and owning your own inference, we we have a a new product on modal to help you like get started with deployment really easily.

s573
00:59:21.119 --> 00:59:29.760
Get optim get an initial baseline of reasonably optimized uh inference engine deployment that you can work off of with modal endpoints.

s574
00:59:29.760 --> 00:59:38.559
Um so check that out and if you're interested in working on deploying inference for hundreds of of teams, uh check out modal.j jobs.

s575
00:59:38.559 --> 00:59:40.271
Thank you.

s576
00:59:40.271 --> 00:59:42.271
[applause]

s577
00:59:54.681 --> 00:59:56.681
[music]