What Is an Inference Engine, Anyway? — Charles Frye, Modal

AI Engineer · 59 min · 577 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:12All right.
00:14Um, so I'm going to go ahead and get started while people get seated.
00:19Um, we got a pretty tight turnaround.
00:21A lot of really wonderful workshops at this event.
00:24Um, so I want to make sure we have plenty of time.
00:27Uh so I prepared for 10 or 15 minutes of questions and I think it'd be best especially given this is a workshop if people just interrupted me as I was going raise your hand and I'll call on you and answer your question.
00:42Um and so just like let's keep this interactive.
00:46Um I think it's it's better that way.
00:48Um, and it seems like talked to a couple of people, it seems like we got a lot of folks here who know their who know their inference, uh, know their engines, and so we can get a a sophisticated
00:58conversation going.
01:00Um, so the title of this talk is what is an inference engine anyway?
01:06And the goal is to sort of peel back some of the layers that are behind APIs and uh what is go actually going on inside of the code that runs your inference
01:19uh when you run it yourself.
01:21So roughly you know there tokens go in tokens come out um and a lot of the time you get to avoid thinking about what happens in between.
01:30Um there's this really high quality open source software out there for running your uh running your language or multimodal model inference.
01:38You've got SGA, VLM, Tensor RTLM.
01:42Many people write similar things to these in-house.
01:45Uh they design these these uh engines that like at a high level just taking in some tokens putting out some tokens is not that hard.
01:56Um you can write something like this in using PyTorch and hugging face transformers in like an hour or two especially if you're good with agents.
02:04Um but making it uh extremely high performance so that you can produce tokens with good tokconomics is the challenging part.
02:12Um and that d like sort of drives the design and architecture of these systems.
02:19Um so just as like a very quick sketch um before we jump back to the high level and talk about why why we're here.
02:26Um the uh these are generally architected as a set of communicating processes.
02:33One that does IO does HTTP or RPC serving and responds to the outside world.
02:40Um and then uh there's a little bit of pre and postprocessing with tokenizers and detokenizers.
02:47uh and this wraps uh the sort of core of the system where the real work gets done um in sort of two pieces.
02:56One a sort of scheduler that like defines and schedules work to run on the accelerator, usually a GPU.
03:04Um and then the actual code that runs on the GPU.
03:08Um and so this these two components in the bottom right here are where the sort of majority of the performance sensitive work gets done and where the complicated problems are.
03:19[snorts]
03:20Um so that's where we'll spend the majority of our time uh discussing.
03:27Um, all right.
03:28So, with that stage set, um, I think, uh, I've been to AI engineer since it was the summit and talked about, uh, running, uh, running inference, uh, every time.
03:42And the answer to this question gets better and better each time.
03:45Uh, the question of like why inference and why you should care about it, why you should think about being uh, an inference engineer who operates inference engines.
03:55um our uh fearless leader and uh uh one-man conference army Sean Wang uh tweeted recently uh that everybody in AI infrastructure the boring stuff uh like inference engines not the sexy research stuff
04:10um is finally getting filthy rich um and so what uh I think he's pointing to here is there's tremendous demand uh for these uh to run inference uh there's demand
04:23at the labs there's demand within companies who want to own their own stacks.
04:27There's demand to set up and rethink the infrastructure stack and all of this wraps this core inference workload.
04:36Um but Sean is also pointing out that a lot of what people have been thinking about up to this point um is sort of the research side, the production of models.
04:44So um why not, you know, why not think about training?
04:48Why not go to the training engines uh talk with your precious time?
04:52I think the fundamental reason why this is such a critical place uh for engineering work to get done and why it's such an like cool place to work as an engineer
05:01is that in the end training is a cost center and inference is a revenue center.
05:07Um so when you train models um it turns out um people have not generally been able to make businesses work where they sell model weights to people directly.
05:16Um and so you can't really turn a training setup into like a proper self-supporting system.
05:22Uh but uh inference once you have a model um is something that people will pay for.
05:28People will pay for a system to you know take in their tokens and their context to produce new tokens, produce software for them, whatever.
05:36Um there's a lot of opportunities to do inference.
05:40we have set up where there are a small number of essentially model foundaries who produce foundation models that are then deployed and customized by inference engineers.
05:50Um and then even for training inference has uh snuck its way in.
05:55A lot of the post-training adaptation of these foundation models requires that you generate samples from the model uh in inference and a lot of the same technologies come to play.
06:05A lot of the same concerns are present.
06:08Um, and then finally, sort of more selfishly as an engineer, there's like right now inference is still very new, even though we're years into this new technology.
06:17Um, and so you end up kind of straddling the whole stack from application architecture uh through linear algebra down to like electrons and hardware.
06:26So it's a an engineer's playground.
06:28Um, so we'll see a lot of that as we go.
06:32Um I'll maybe in the interest of time since we kind of had to get started very quickly I'll skip over this but I've been working on inference for a couple of years mostly
06:39uh at modal um since coming from the sort of training world before that um and helping a lot of teams sort of deploy and optimize their inference on hundreds or thousands of GPUs on the modal platform.
06:50Um I've also been making a lot of uh inference applications myself.
06:54This is a little art piece that I made.
06:56Um, we uh uh had it in the uh Legion of Honor just this past week.
07:01Um, this art piece takes what it sees, passes it through a vision language model, one of the Quenes running on uh uh served via the SGA inference engine and running on modal.
07:14Um, it takes that input and it describes it as though it were an art piece.
07:17So, it produces one of these little museum placard uh type things.
07:21And as everybody knows, you're not art until somebody writes one of these little placards to describe what you've done.
07:26Um, so I pointed it I pointed it at the line on the way in.
07:29Um, and it came up with a cute little description.
07:31Anyway, so this is the kind of, you know, uh, sort of in the middle of the set of inference applications between the lowest latency and the uh, highest throughput applications that get deployed.
07:43Um so uh I'll start by talking in more detail about those workloads like what workloads land on these inference engines and then jump quickly jump move from that quickly into architecture
07:54how these engines are designed some of the key techniques uh that fit within that architecture to make their uh improve their performance.
08:01Um and then lastly, a key piece for making these things work in production, observability and uh like debugability of these engines for correctness and for performance.
08:14All right, so let's start with workloads really quick.
08:16Um uh I like to break things down into kind of three pieces.
08:20There's chatbot plus, which is like the original killer app of inference, something like chat gypt or clawed code.
08:26there's a person waiting on the other side interacting with this inference system and providing additional context.
08:32Um the plus uh indicates that not only does it talk to a person, it also probably interacts with external systems via tool calls.
08:40Um then you have background agents.
08:42They often do similar tasks to something like clawed code.
08:45Um but they do it in the background, not directly in response to a person.
08:49So something like Devon, Ramp inspect, uh OpenClaw or Herbies for a non-coding case of a background agent.
08:56Um so uh this is uh characterized by a similar sort of affordances or workloads but a different performance profile.
09:04And then lastly data processors.
09:07Um these generally run relatively small models.
09:09They take unstructured data like a PDF and they turn it into structured data.
09:14Um they extract some information from it.
09:16Um usually the sort of thing you might insert in a database.
09:19This is something like the uh reductto document processing platform or something like Fathom that takes a a video and extracts the transcript.
09:30Um okay so under the hood what do these uh workloads look like?
09:36Um you send in some tokens from a client uh that goes to some server somewhere.
09:41Um for now we're going to focus on cloud deployment of inference because that's where the majority of it happens right now.
09:46Obviously in the future a lot of these inference engines are going to end up running on people's uh machines like in a couple of years.
09:53Um yeah, go ahead.
09:58The size of the screen
10:01brightness of the screen.
10:03Um yeah, so the question was about adjusting the brightness.
10:05I'm not sure if we can uh fix that.
10:07That's what I get for making dark mode slides, but yeah.
10:11Um yeah.
10:13All right.
10:13So a client submits um an input uh and then it's processed by our engine running on a server somewhere.
10:20Um first we process the input tokens in the prefill phase and then we generate the output tokens in the decode phase.
10:27Um, and so we have uh one uh one prefill uh run uh does a forward pass through the model and then multiple decode runs to produce one or a few tokens at a time and then once we're done return them to the user or generate a tool call and interact with an external system.
10:48Um so with these work these both the external interface and this sort of internal pre-filled decode split defines these uh the sort of metrics that you care about for a workload that sort of tells you how to deploy your inference engine and and how to measure its success and performance.
11:06So you have something like queries per second in and queries per second produced.
11:11Um this one pretty tricky to think about uh under control of your users.
11:15um uh you probably have a better time thinking of it uh at the engine level thinking of how many queries per second you can handle per replica of your inference engine.
11:24Um and then you have aggregate demand coming from users.
11:27Uh then with those queries you want to look at how many tokens there are per query um input and output aka prefill and decode.
11:37Um so this one is tricky because it's not only under control of users but it's also under the control of a model.
11:43Uh so the determination of when to finish processing a query is not under your deterministic control.
11:49It's under the control of the model inside the inference engine.
11:52Um so this is something not both of those sources of nondeterminism means you need to kind of like measure this benchmark it to get it.
11:59Um and then uh when we do our processing we uh we produce some intermediate artifacts that we want to cache in the KV cache.
12:08Uh and then the degree to which that cache is reusable is a key component of the definition of our workload.
12:15Um this is also mostly under control of users but you can get like depending on the workload there's actually like pretty uh relatively fixed patterns of of prefix reuse.
12:25Um uh and then finally coming from the application layer um you have some uh latency budget some SLO of like how quickly you want to return tokens.
12:36That could be how quickly you finish something like the prefill phase and have your first tokens ready to send to users.
12:42Or it can be how quickly you can generate each set of tokens during your decodes and get them back to users.
12:48So that's your time to first token and your time per output token or inter token latency.
12:53Uh so this is going to be the like primary constraint that you operate under.
12:58Okay.
12:59Um, and we can map those back onto our application archetypes.
13:03Uh, like chatbot plus background agent, really high prefix reuse.
13:07There's some interactivity going on whether that's a person or an agent.
13:11Um, the chatbot plus generally has relatively short decodes, relatively short outputs of tens to hundreds of tokens.
13:17Um, and a very tight latency tolerance on that.
13:20Like a person uh wants to see something in a few hundred milliseconds at the very most, or they get bored, uh, they get angry, they turn away.
13:27Um, with a background agent, you maybe have a few minutes to produce a PR, right?
13:31Um, if it's as high quality as an engineer, maybe as like what would be produced by a good software engineer, maybe you have a few hours.
13:37Um, and so there's not as tight of a of a budget on your milliseconds of network overhead or whatever.
13:43Um, and then with a data processor, you generally have very low prefix reuse.
13:47Um, you are seeing different documents each time.
13:51There's maybe a system prompt, but it's generally small relative to the other inputs.
13:55Um, and then you have a very short decode.
13:57you're producing some structured object that's generally short relative to the input.
14:01Um the structured version is shorter than the unstructured version.
14:05Um and then the demand here is less on latency generally and more on aggregate throughput because you're generally like ripping through a database backfill type job in a lot of these.
14:17All right.
14:18Um and so maybe just want to call out uh like now that we've looked through this you could see that like even though at the application layer this feels like kind of one it's one single workload from the perspective of the user once you start drilling down into what the engine is doing it really feels like there's
14:36two subworkloads going on.
14:38there's the like workload to do your prefills to do the processing of input prompts.
14:43Um, and then there's work to do the decodes to produce the outputs.
14:47Um, and these uh they sort of operate semi-independently inside the engine.
14:54Um, and so need like a sort of like different perspective and they follow often like different code paths.
15:00Um, and they're sort of defined by the fact that the per user in the decode phase, you do you're doing far less arithmetic, far less math per uh per decode per user than you would do in the prefill phase, which is a lot like much higher arithmetic intensity, a lot more uh uh a lot more math per
15:21request.
15:22Um, and so that's your fundamental constraint, but also the sort of space in which you operate in configuring an engine and then like adding features or um, adjusting the engine behavior to serve these two things.
15:34Okay, so I wanted to make sure we're all on the same page about some of the core stuff about what these engines are actually doing before we peel open
15:42uh, the insides and start looking at them.
15:44So what is the architecture of an inference engine?
15:47Um so um this is a little little diagram showing at a high level what an inference engine does.
15:54So the goal of an inference server is to take an HTTP request or a gRPC request and turn it into an HTTP response to the client.
16:03Um and so it does that by passing through all these things down here.
16:07Um so the idea in this diagram um is that if you follow any set of arrows you get the same thing.
16:14So uh you can sort of peel back a layer of the arrows and look inside and say all right so what what is the the inference server does this high level thing what's the next level down the inference engine is
16:25the sort of like interesting part of it is the is the part that is not doing the HTTP serving because we aren't in one of those situations where HTTP serving is the hard part is your bottleneck it's not like a proxy
16:36um it's uh it's a this is a case where you're handling tens hundreds maybe thousands at most of requests and responses per replica and so general purpose hardware just you know kind of eats this for lunch.
16:50Um so the real interesting stuff is all at the layer below this.
16:54Um so at the top is your like your server IO for HTTP requests.
16:57At the bottom is the actual engine and it's like collection of processes.
17:01Um and you can kind of break this out into three pieces.
17:04The request comes in expressed using like sort of human friendly um or external representations.
17:12So you have Unicode strings, you have PNG images, you have MP4 videos or WebRTC streams.
17:19Um, and these need to be pre-processed before they can get passed to the actual inference.
17:23So they get turned into collections of tokens.
17:26um which then get mapped into tensors on the GPU and then we can actually do the inference part of the inference engine which takes you know in the simplest case it takes a tensor
17:37with K entries in it and gives you a tensor with K plus one entries in it and these are machine learning models that predict the next token they guess what would the next entry in this big tensor look like
17:49um based off of the data I've seen in training whether that's data from the internet or data from what a helpful agent would do or RL data um whatever that you learn to predict this next tensor entry.
18:02So that's the core right there.
18:04Um and that is what is wrapped by the rest of the inference engine.
18:07Um so then we have post-processing on the side here.
18:10So post-processing turns that new tensor into a response.
18:14Uh and that's generally done by this detokenizer that goes back to the sort of like token space first and then uh and then back to our um back to the human and external system friendly representation in the response so it can be returned to the user.
18:31question.
18:34Could you explain the difference between the inference server and the inference engine and where does
18:41Great question.
18:42Yeah.
18:42So the question was what is the difference between an inference server and an inference engine and where does something like VLM lie?
18:49Um so inference server uh means that you've wrapped some like input output around the core operations of like taking requests and turning them into responses.
19:00Uh so VLM has both of these like VLM operates as a an HTTP or gRPC server.
19:08Um and then you the engine layer is sort of like part of the VLM or suing SDK as like VLM.
19:18Um so most deployments are look like are deployed as a server with like a single request single response kind of interface um as depicted here.
19:29Um yeah, does that answer your question?
19:32Yeah.
19:32Um cool.
19:35Um I said this earlier but some people have come in since like please interrupt raise your hand with questions if I'm like if there's anything confusing or if you'd want to dive deeper.
19:44Yeah.
19:44Rom.
19:50Yeah.
19:50So the question from Rustom is is this like a stateful or stateless uh workload?
19:57So the answer is it is stateless
20:01[snorts]
20:01if you don't care about performance.
20:03Um it's stateful once you do care about performance which is pretty much every case.
20:07So and the primary state that you accumulate over time for requests is the KV cache of like previously computed things for for a previous request attached to some notion of session.
20:20um and that handling things like sessions actually lives outside of the inference server/inference engine in general.
20:27So the an individual engine replica can track this KV cache but is up to the wrapping layer which might be something like Nvidia Dynamo or LMD to handle like
20:39defining what a session means.
20:40So it's like the clients in that architecture sort of define the the sessions.
20:44Um and that is where a lot of the trickiness comes in.
20:47Um, so we won't talk too much about that because it's kind of outside the there's enough to talk about with engines um already, but yeah, uh, come find me afterwards
20:54um, if you want to talk about that layer.
20:58Um, right.
21:00So this diagram, we also talked about it previously, but I just want to return to it now that we've seen a little bit more.
21:05So this outer layer is our server IO process.
21:08This um, is this is specifically SGA's architecture.
21:12There's some slight differences with VLM, but this server IO process communicates with the outside world and then with our pre-processing and post-processing.
21:21Those things communicate with auler process um which is the one that defines and schedules work onto the GPU, handles things like queuing.
21:29Um and then the not ready for that yet.
21:33Um, and then the one or more processes that are sort of closely tied to the GPU are the ones that do the real work of doing the inference.
21:43Um, so the yeah, so the key things to pull out here is that if you are an old school ML person who's been in this world for a long time, the model forward passes layer is the one that you're familiar with.
21:54This is where the vast majority of the sort of like PyTorch uh and similar code shows up.
22:00Um and this is the pl like the GPU or the accelerator in general is the component that costs the most um and is capable of the most work like paflop
22:12per second scale operations.
22:14So this is where the like hardcore shave a shave a microcond performance engineering is focused.
22:21Uh but critically thisuler process out here or it's equivalent in another architecture is your is the bottleneck to that component.
22:31Um and so this is it's generally operates um it is literally singlethreaded in implementations we'll talk about but is also like semantically it's single threaded.
22:41It's this one thing that is defining somebody has to define the work that happens on the on the GPU.
22:46somebody has to manage a bunch of resources there and there's some some fiddly like tricky things to do concurrently.
22:53And so this becomes this uh like semantically singlethreaded uh like point of control that can potentially despite the fact that it's not doing much it's not doing as many operations
23:07uh as the work on the GPU it actually becomes your bottleneck.
23:11It's the sort of gatekeeper to the GPU.
23:13And so this is the other place where the majority of like performance engineering and care needs to go in the design and operation of an inference engine.
23:22Um so we'll see more on that as we go but but that's like the high level takeaway about these things.
23:28So is the solution to that bottleneck problem?
23:38Uh yeah so the question was is the solution to that bottleneck to you know operate the schedulers in parallel or um uh or to improve some of the external components that consume from it.
23:50I think the current state of affairs is that this is not reached the like that you can operate these server processes in Python and you just have to be faster on the host side than the GPU side
24:07and there's not the like complexity increase that you get from trying to manage like locks or coordination on the GPU resources is much that complex lexity increase is much greater than the gain
24:22um because even if you were able to get it down to a microscond the work wouldn't start happening until the previous work on the GPU had finished and that's
24:32you know depends on the particular workload but you're talking generally tens of milliseconds to hundreds of milliseconds and so the like um yeah there are general most of the time you can get around it we we'll talk a little bit more about this so maybe ask again once we get to if if that wasn't satisfying.
24:51Um but yeah, great question.
24:54More good questions, please.
24:56Okay.
24:56Um so then just looking at that was like the architecture diagram that you think of as your like engineering.
25:01Um let's look at the like life of a request in this architecture.
25:05So the um the server like receives an input from the outside world.
25:10It says okay, we've got a request.
25:12um it sends it to the tokenizer to prepare or pre-process this request that then goes into the scheduler that decides like okay how many resources is this thing going to consume what resources are available
25:24um and then that runs this core loop of creating um batches of requests to go in the model runners.
25:33So generally most of the accelerators like benefit from operating on multiple requests in parallel.
25:39Um so especially GPUs but also TPUs and and many others.
25:44So you want to collect a bunch of requests together and run them.
25:47Um if you're a database person you might imagine like I'm g I'm about to run a sequential scan so I might as well grab like three or four queries that are all going to like read this entire table and do all of their um like you know do all their queries at once.
26:01Um because the hard part is that like sequential scan of loading the entire table.
26:05Um so there's a similar thing going on here.
26:08Um uh so this core loop operates over and over again start a batch and outputs the like uh the tokens and the probability distribution over tokens the the logits.
26:18Um that comes from the the model runner or runners.
26:22Um so that operates um sort of continually.
26:26Uh and then as tokens are ready uh they can be passed to the detokenizer to be like turned back into a proper response to the outside world and then sent back to the server.
26:41So yeah, this is what's interesting about this stuff is that it is simple relative to what's going on inside databases, file systems, uh like web servers.
26:50Um, partly because it's fairly new, but partly because there's just this core bit at the middle that you just want to operate at maximum speed.
26:57Yeah.
26:57Question.
27:03The question was, is there communication between the tokenizer and the detokenizer?
27:07Um, to my knowledge, there is no direct communication between the tokenizer and the detokenizer in SG lang.
27:14Um, in VLM, they actually live inside the same process.
27:18the um manager, the engine manager.
27:21And so like it'll be a little harder to determine if there's any communication there.
27:25Um but in general, they actually don't need to communicate with each other.
27:28They just need to operate essentially the same um system for mapping from like inputs to uh tokens or tokens to outputs.
27:35Yeah.
27:36Did that answer your question?
27:38Yeah.
27:47I'm not sure
28:06Right.
28:06So the the question was about basically maintaining coherence between the tokenizer and the detokenizer and how would you handle maybe like ambiguous um like ambiguous inputs.
28:16Um the answer is that yeah they they run the same like underlying li like underlying libraries like the usually like the tokenizers library from hugging face.
28:26Um and the I would say the tokenization part is the one that is harder than the detokenization part.
28:35the detoization part is like a little bit more entirely under under the control of the person implementing the the detokenizer.
28:42Um yeah.
28:44Um
28:46software
28:50yes.
28:50So the question was does the software allow the tokenizer and detokenizer to always make the same decisions?
28:55I would say yes.
28:57I'm trying I'm like trying to make sure that I'm not overpromising here.
29:01Um but I don't think you need communication between them to ensure that the same decisions are made.
29:05Yeah.
29:06Question.
29:06Sorry, the person.
29:07Yeah.
29:12Yes.
29:21Right.
29:21So first pointing out that the tokenizer can be um parallelized and generally is like multiple processes um and yet you still see longer time to first token uh if you have longer input sequences.
29:35So why is that?
29:36Um so the answer is that processing a longer sequence of tokens takes longer inside of this core loop.
29:45Um so it's the like primary driver.
29:47Um if you have a 100,000 tokens that will take let's call it 10 times 10,000 tokens.
29:53Um once you get to a certain uh size it's basically linear in the input sequence size.
29:58Um then in a like running inference engine there's also effects from like you break up the input tokens into multiple pieces and run them as separate batches.
30:06So you can also get additional if there's like queuing and congestion that adds more to the um uh processing of a uh of a long batch um than to a short batch.
30:19You experience the queueing delay multiple times possibly.
30:22Um but maybe the one underlying um point to clarify like producing the first token means producing the first output token not the tokenization process.
30:32Um and then finally not a part of the question but something I wanted to point out.
30:36It is interesting that you generally have multiple deto tokenizer processes um and one dtokenizer in the sgang architecture at least.
30:44Um and the answer there is that input tokens are come in at like a much higher rate than output tokens are produced.
30:51Um and that's related to the fact that prefill can operate at much like higher rates of total tokens per second than decode can in general.
31:00Um, and so you there isn't as much pressure on the detokenizer component of the system to operate like at the as quickly.
31:06Um, and it's also in some it's bottlenecking your response to the user, but it's not bottlenecking the core model forward pass resource.
31:14So it's also like off the hot path a little bit.
31:18Um, yeah.
31:27Yeah.
31:27So the question was how much of the latency comes from this part in the tokenizer versus this part in the core loop.
31:32This is going to depend on your workload.
31:34Um I would say there was a nice thing from uh who was that Cruso I think um that pointed out that the existing tokenizers generally don't show up as a bottleneck
31:50if you have models that are let's call it like 20 billion parameters total or or above.
31:57um and that and if you but if you are operating a smaller model and you're operating on very large input contexts then it can show up as a as a bottleneck.
32:06So that's maybe a good rough guide.
32:08I would say like I actually have not had to think about the tokenizer latency very much.
32:14It's generally been prefills and substantially decodes that are your the like latency bottleneck.
32:20And so decodes if you're producing 1,000 output tokens a second, each one is like one millisecond on average.
32:27Um and the um but up to like 10 milliseconds, 20 milliseconds um or even more if you have a um lower decode throughput like a large that you're running at lower interactivity.
32:41And so that very the tokenizers are definitely millisecond scale or less and only happens once um per request.
32:50Yeah.
32:51Also notable that nobody talks about tokenizer caching.
32:54People talk about KV caching but not tokenizer caching even though you could do it.
32:57Um I'm going to keep going just to make sure we get through more stuff but these have been great questions and so if it's still relevant please ask.
33:04Um another important question to ask wait why are we doing multipprocessing?
33:08The key reason is they want we want separate threads of control for each subcomponent for our server IO for our tokenization and detokenization for each model runner.
33:15Um we want them to be able to sort of operate as independently of each other as possible.
33:20Um they are many of them the tokenizer and the detokenizer are doing relatively computensive work relative to most of what happens maybe in a in a in a web server.
33:31Um the model runner is um like interfacing with the GPU or accelerator.
33:38It interacts with that in a way that's kind of similar to the way you would interact with a nick um in like a web server, which is to say you offload work onto it.
33:46Um and uh so that one you you're giving it its own event loop.
33:51Maybe in theory that could be part of like a broader event loop.
33:54Um uh but it's uh certainly like cleaner to have these separate threads of control and optimize their uh you know optimize their latencies and there's not sufficient uh interprocess
34:06communication to require them to like say live in the same thread.
34:09Um but why not threads uh for each of these instead of processes um and one of the fundamental reasons is that Python is single threaded per process.
34:19This has recently started to change.
34:20there's still a lot of work for compatibility and performance to be done there.
34:24Um so that uh because Python has a global interpreter lock for all the threads in the same process, you would like frequently block on that.
34:32Um so it's so that is another reason to sort of split these up.
34:35Um but that just uh begs the question um uh or I guess begets the question why Python um and the answer is Python has excellent model or GPU support.
34:47Uh so it's like easy to run code on accelerators with Python.
34:50So specifically for that model um uh model for pass component um and so wait but why wait why does Python have excellent support here?
34:59Partly it's that it's a great language for researchers because it's like ease of use and these uh models are still coming out of a sort of researchy data sciency environment.
35:08But there's a deeper reason which is that Python has really has kept like really deep C and C++ interop as one of its core design principles and that allows for like much easier communication
35:19uh with these accelerators and uh with like accelerated code in general um than uh like a maybe equally ergonomic language like JavaScript.
35:29Um but why not like rewrite it in Rust?
35:32Um the model worker processes again they're just trying to like tell the GPU what to do.
35:38So you could like the code on the GPU is written in a compiled language and and like carefully optimized like to within an inch of its life.
35:45Um but the model worker processes are just organizing that and there's a few tricks like CUDA graphs um that like really just pull all the work out of the worker processes
35:55and into sort of fast stuff running on the GPU.
35:58Um but there is theuler on the other hand.
36:01So theuler just needs to stay faster than that work.
36:04Um, but as GPUs get faster and as the amount of work on the GPU goes down, the pressure on theuler component goes up.
36:11Um, so this is actually like a pretty good target for a rewrite it and rust type moment.
36:15Um, because this part is host side.
36:17It has to manipulate a small number of GPU resources, but it doesn't necessarily have to know stuff about models and pietorrch.
36:23Um, and so it's like a thinner rewrite.
36:26It's like specific per engine.
36:29um and uh uh and host side only.
36:32So like in the sort of arc of future inference engines, I'd expect it to be a big target um making that faster now that the the sort of iterative development speed is slowing down a bit.
36:45Okay, so now let's look inside of the like what what is going on inside of these components.
36:51Um um mostly mostly focused on the um the model forward pass piece.
36:58So at the very we're going sort of like inside out.
37:00So at the very bottom there are kernels that run operations for as part of the model architecture in general.
37:07Um these are mostly deferred by the engine which is to say like the engine doesn't implement all of the kernels for every operation it needs to support.
37:15Um it has backends that implement the uh these kernels and it consumes them from external libraries of kernels.
37:22This is one reason why there isn't much differentiation between the engines on sort of raw GPU performance.
37:28Um, it's mostly about how good are you at getting out of the way of the GPU.
37:32Um, there are a few special kernels per engine like tree attention for treebased speculators and SG lang.
37:38Um, and so there's like little bits where an engine will have a specific kernel, but like if it's useful, if it's popular, it ends up in some kernel libraries so it can be used in lots of places and then they consume it from an external library.
37:51Um then uh the kernels are selected by this sort of model forward pass code that does control flow of and sort of like selection of which kernels to operate.
38:02Um and this is this is implemented in each engine in general and it's implemented per model like if you want to add support for a new model architecture this needs to be implemented in each engine.
38:13um there's generally a reference implementation that comes from a library like transformers um but that is insufficiently optimized um the like the setup there.
38:23So there's um I guess in suang and vlm there is now a path where you can just use that transformers implementation and they'll maybe do a little bit tiny bit of surgery to see what they can automatically
38:33kernel fuse or automatically optimize or rewrite.
38:36Um but in general for most things people want to run you're going to go through this hand handwritten as handwritten as software is these days uh path for um like selecting which kernels to run.
38:48Um then uh wrapping those model forward passes is our batch construction to select the inputs to that model forward pass.
38:57Um this is kind of one of the primary sources of special sauce per engine.
39:03you need to set up this um this sort of like queuing setup.
39:06Um so you might want to be able to for example not operate separate prefill and decode cues but operate one sort of queue where you can mix prefill and decodes together.
39:17Um that might have some performance implications.
39:19So you might also want to be able to like dynamically switch whether you're running pure prefills, pure decodes or mixes.
39:25Um and then you want to do this I think in the ideal operation of an inference server there is no queuing.
39:30Um, so as somebody who operates these things, I like I treat queuing as a problem.
39:35Um, but the engine should be able to kind of do it.
39:39Um, or may maybe it's too extreme to say that queuing is a problem, but um, you don't want this to be a like yeah a growing queue.
39:47Um, you want it to be flushed as quickly as possible.
39:50Um, and then lastly wrapping that you have uh detokenization and tokenization.
39:55Um, this is like mostly boring I'll say.
39:58um it's like solved in the existing tokenization libraries outside of the engine.
40:02I think the exception to this being multimodal inputs and outputs um we now have pretty good open weights multimodal input models.
40:10Um and if you take a look at some traces you'll see that there's like still many operate opportunities for optimization on this path.
40:17Um, tokenization of an image is just like a harder problem than tokenization of a um of a string of Unicode bytes like conditioned on already having a a tokenizer.
40:29There's more more work to be done.
40:31You might need like ffmpeg for instance to like pull out raw frames to go in to something that turns it into a tensor.
40:38Um, and so yeah, so that is maybe the most interesting part there.
40:42Um the one way to sort of conceptualize the structure around the model forward codes is that the model forward code is um like going to call into a couple of different types of libraries
40:57or directly sort of write its own code to run on the GPU.
41:01Um so a lot of it goes through like PyTorch code which has its own kernel libraries and and can run things on the GPU.
41:08Um then there are like specific kernel libraries that you would draw from that aren't in the engine.
41:13And then finally the engine has its own uh set of like kernels that can directly um you know run code on the GPU.
41:22Um so picking that apart a little bit the um uh the kernels that exist in their backends.
41:29The core of language modeling architectures these days look something like this.
41:35Um there's a sequence of layers uh that you take an input, you pass it, do some cross token computation on the sequence, then you do some per token computation to enrich the sequence.
41:47Um and then you merge that back in with your representation and you iterate this layer after layer.
41:52Um and so this splits nicely into the two kind of classes of like really critical fast kernels that are available especially via the backends.
42:02Um so the cross token computation is your attention or linear attention.
42:07Um there's a bunch of different backends for attention.
42:10The flash attention kernels from tree and jaw and others.
42:14Uh flash and fur from Nvidia.
42:16Uh cutless also from Nvidia.
42:18Um the Triton uh implementations of these kernels.
42:22That's OpenAI's kernel authoring library.
42:25Um, the TRTLM, the engine has also split out its kernels and they're available um, as as backends for attention.
42:34I think you'll see many of the same names that show up as attention backends also show up as backends for your matrix multiplications inside of your um, your uh, per token computation which is your MLP layer.
42:48It's where the biggest uh, gems are.
42:51Uh it's also in some architectures it's this is a dense MLP a classic neural network.
42:56In other architectures it's these sparse mixture of experts which basically just says don't send every token to the same neural network.
43:03Per token decide which smaller neural network to send it to.
43:06Um so the um the dense MLPs there's a there's a nice uh library for this from uh deepseek deep gem that gets used a lot in sglang as kind of like the default.
43:18Um in VLM there's some quantization focused backends for this the Marlin and bits and bytes backends on the uh for the group gem you can split this out into
43:32first like there we have this like routing or communication problem of where do tokens go and how do I bring stuff together at the end.
43:39Um so that can be done with a separate set of kernels like the uh for this all to all communication and routing like the another one from deepseek DP.
43:48Um and that could be sort of factored out from the like matrix multiplication step.
43:55Um that had that can have performance penalties and so you can also do that as like one big kernel um a mega kernel uh which is what is done in the deepseek v4 architecture.
44:07Um okay so these uh with VLM and SGA lang these are things that you can sort of set and control in SG lang it's like kind of pushed up into your face in the form of configuration at start with smart defaults
44:21VLM is a lot more focused on the like smart defaults side and then you sort of like patch change things more with um environment variables.
44:31Um, at least last I checked, you know, these things change every like few days.
44:35Um, but yes,
44:37questions are like who decides how much?
44:58Yeah.
44:58So the question was basically about what's happening with concurrency with really large numbers of requests.
45:04Um so that goes back so that's uh much higher level than the kernels and backends.
45:09Um that's happening sort of at our like batch construction and scheduler level.
45:14So basically the scheduler is going to do what the best that it can to decide like um which requests to run emp generally emphasizing requests that already have KV cache allocated to them um because then you don't need to recomputee
45:28um the if you need to evict something from the cache to make space for another request um yeah that will generally get like put out to CPU memory and then disk
45:39um and I would say Yeah, to my point earlier about queuing, I think of that as essentially like a failure state of the engine.
45:49Um, that we like I've deployed my engine improperly.
45:51If I get to the case where I have KV cache thrashing or like things being flushed to disk in the same way that there's like swap memory in your operating system, like you can swap memory to disk, but at that point your MacBook becomes unusable,
46:03right?
46:04Um, so it's it's a useful feature to have, but almost like um so that you don't go hard down when you have a serving problem.
46:12Um yeah, how is mobile?
46:29Yeah.
46:29So the question was about like serverless deployments and cold starts.
46:32I'm not going to talk about that this much because I promised Swix I wouldn't do a vendor talk.
46:35Um but please talk to me afterwards.
46:38There's a blog post called truly serverless GPUs that talks about how we solve that problem.
46:43Memory snapshotting, GPU memory snapshotting, custom file system, bunch of pieces to make the cold starts faster.
46:48Um, yes.
46:50Um, cool.
46:52All right.
46:53So, let's talk about some of the key techniques that you need uh to implement to in inside an inference engine to make sure like uh your performance is good.
47:03Um so attention is fundamentally quadratic at least like the uh loadbearing part of the attention in in every architecture that's popular.
47:12Um and the um so the solution to that is to store things um after you've computed them and you do like a little space-time trade-off for linear computational complexity in exchange for linear storage.
47:26um that runs into problems with how much can you store in GPU memory because it's actually often slower to store this somewhere else uh and then load it back in as opposed to recomputing it.
47:36GPUs are fast.
47:38Um so you want to be efficient with your use of GPU memory.
47:42Um and then you often have requests that share prefixes.
47:46So for example like all these commandments start with thou shalt not.
47:51Um, and so you can reuse the computation of everywhere up to here.
47:55Um, as soon as a single token differs, you no longer can share the prefix.
48:00So it's kind of this like tree or try shaped structure.
48:03Um, but you want to be able to make use of that, especially if you have like a giant system prompt for like an agent system that's tens of thousands of tokens.
48:11Um, it's like quite nice to be able to share that across users.
48:14Um and so the solution to this is like KV caching with pages or or radices.
48:21Uh so you store the information in this um in like a a page cache data structure like you'd have in an operating system and then you just need to reconstruct these tensors sort of on the fly to pass them to attention kernels.
48:34So that that job is now it used to be the job of the engine to solve that problem.
48:38That's now incorporated into the attention kernels like flash attention 4. something that our team worked on the um page size one um and sort of like RAD XC form of KV caching you can see on the right
48:50um and uh yeah so this is this has moved a little bit I put this in here because it used to be like a defining feature of the engine but now it's kind of actually part of the kernels and really it's the management of KV cache capacity and um and the like layout of the KB cache that's
49:07the problem of the engine um another problem I already talked about how this the scheduler can be this um host side bottleneck.
49:15What you want when you're running stuff on the GPU like both at the sort of level and then inside of models is that you want that just like at every moment some kernel is running on the GPU and things happen on the CPU to decide what that
49:29uh should run on the GPU.
49:30But they shouldn't ever block uh progress.
49:33But it's very easy to accidentally write some kind of host device sync.
49:36There does have to be information passed between them.
49:38the GPU can't just like send the response directly to the user.
49:42Um, it wouldn't be that good at it if it did run an HTTP server.
49:45Um, and so you you do need this CPU work.
49:48You just want to make sure it doesn't block.
49:50Um, one of the key techniques for this is CUDA capture.
49:54So you take all of these kernels that get run in the forward pass of the model and you track all of them and you turn that into one big data structure a DAG of operations
50:04of like this kernel runs and and like you know mutates the data at this pointer.
50:10Once that is done this kernel can now run to mutate what was present at that same pointer and mix it with data from another pointer.
50:17So this big DAG of C of of uh operations is your CUDA graph.
50:22And this um allows you to go from launching like each one of these GPU kernels having a little bit of CPU work to decide what to run, how to run it, uh what you know which pointer to put
50:34um you turn it into just one CPU side work to decide which graph to launch and then all of it gets launched from that single graph which is what's depicted in this um
50:43uh peretto trace.
50:45Uh so this allows you to relax quite a bit about what is going on in the model uh forward pass component at least.
50:55Um another problem decode here is sequential um and every time you run it you have to load like either all of the parameters of the model or all the active parameters in ane
51:06architecture um and you need to do that over and over again um by default for each token.
51:11So you're like on behalf of your users, you might be loading like a terabyte of data or 100 gigabytes of data per token and you want to produce a token every couple milliseconds.
51:21Um so now you're looking at that's pabyte per second scale uh memory bandwidth.
51:27That's uh pretty hard to achieve.
51:29Um so one solution is key technique for solving this is speculative decoding where you take um a separate model.
51:36The speculator guess what the next couple of tokens will be and then pass it through the target model.
51:42Um and then the target model sort of grades all of those in parallel.
51:46Um and if you apply the right sampling techniques, rejection sampling, um then you can uh guarantee that uh the output from the target model is the same as you would have gotten if you'd run it sequentially
51:59up to numeric um the same output uh but you've now run it in parallel.
52:04You sort of turn your decodes into tiny prefills which is nice.
52:08Um and so the um I'll skip over a little any details about the algorithm.
52:13There's a great we wrote a blog post about this uh just this past week or two.
52:18The key reason why this is really important is that as you improve the quality of the speculator, as you improve the number of tokens it can guess in a row,
52:26um you actually get basically a linear speed up in your decode throughput.
52:31So like many other optimizations that you can do on your inference engine are like the it's like a game of inches of like a percent 5% 10% like each time and you have to do the like very like difficult performance engineering
52:45for each of those you know percentage points.
52:47Um but then on the other hand you could sit down and train a better speculator model um and to first approximation throw data and compute at it um and then get a 2x speed up or a 4x speed up or an 8x speed up um on top of the the the baseline.
53:06Um so this is has emerged as one of the like key techniques to allow you to operate say like a thousand tokens per second on um on Nvidia hardware without have for actually fairly large models
53:18uh without having to go to um like a a novel accelerator like an LPU or Cabris.
53:26Um I'm actually going to there's a lot of parallelism techniques but in the interest of time I'm going to kind of skip over this one.
53:32um you want to be able to split work onto accelerators.
53:35Um a lot of this comes from the underlying kernels.
53:38You need good kernels that can operate with this parallelism.
53:41But within the engine in the sort of model forward passcode which is written on a sort of per engine basis um you need to do all of the communications
53:50around that.
53:51Um and so the communications is where a lot of the problems come in in these uh parallelism uh techniques.
53:59Um all right.
54:00So, uh, observability with our last couple minutes here.
54:04Um, inference services like all services will have some bugs in them.
54:09Um, you have everything from application level bugs from models misbehaving.
54:13Um, uh, which is really not the engine's problem.
54:16It's the app developer's problem, but they'll show up on your dashboard sometimes.
54:20Uh, model quality bugs.
54:22These are the problem of the engine.
54:23Are you actually doing your inference correctly?
54:25When you applied this performance optimization, did you actually guarantee it didn't change model behavior?
54:29Um, one of the most common ones here is actually on model release.
54:32Tokenizers are often slightly bugged.
54:34They both tokenize and apply chat templates and there's often some like uh some issues there.
54:40Um, they're beaten down in a couple weeks after model release, but it's something you always have to be like on guard for.
54:45Um, and then performance bugs or engine performance issues.
54:49The kind of the worst ones are regressions over time.
54:52as you operate a replica for an extended period, you might see performance degradation from issues that only arise once the thing has been running for an extended period.
55:00Um then you also might observe, especially in a heterogeneous cloud deployment environment, differences across replicas in their performance um that then need to be investigated.
55:10So the goal with these as with all kinds of like production debugging is observability.
55:15Um if you can log enough information uh to debug just from those logs, then you don't have to spin up a development server and take a bunch of extra time.
55:23Um and so this runs from application level stuff um to I think focusing on the engine problems, model quality.
55:31You'll you want to be able you want to be good at running evals for model quality as part of your deployment.
55:36Um so you want to hit your actual deployment with like evaluation benchmarking type scripts.
55:43Um there's scripts in SUAN, there scripts in VLM, there's external benchmarking tools.
55:47Um take those uh calculate some numbers, log traces somewhere, log as much data as you can about those so you can debug later.
55:55Um and then in prod log traces and try to get feedback from users on correctness that you can thread through all the way back to those requests.
56:03Um for tokenizer bugs, hot tip, log the token IDs.
56:06Set your engine up so that the token IDs are logged as well.
56:09Um and then finally for performance, performance bugs can show up anywhere.
56:12So the more metrics that you log uh the better like more metrics than you think you need um because you never know which crossorrelation will re will make the bug super easy to discover instead of super hard.
56:24Um so rather than going through those I'm going to talk a little bit about this.
56:28This is an endpoint dashboard on modal and what like how you might read it.
56:33Um I was operating that um the uh art project uh that I showed at the beginning and I just actually spun up like three different replicas of it uh to see what would happen like under increased load.
56:45Um so there was a traffic spike which you can see down here that turns into increasing uh first time to first token as things start to cue uh on the prefill side and and don't enter prefill
56:56um until uh the things have finished.
56:59uh and then also shows up as well as inter token latency as uh decodes get first get a little bit slower and then a lot slower with this queuing as there's sort of contention on the underlying GPU.
57:12Um and so the solution in this case with going back to the question about serverless deployments is this is detected as like increased load.
57:20You spin up new replicas.
57:21Those replicas come online.
57:23They start taking on some of this request load um and then the um like the amount cued um uh decreases back down to baseline and things are um you your congestion is solved.
57:35So I think um that's the like primary way.
57:39I think in many cases, you know, we scale up additional resources to solve congestion and queuing.
57:45Um, so yeah.
57:47Um, when you need to go deeper, I'll skip over this.
57:50Um, but there's the NSI systems or torch profiler is one of your key tools here.
57:55It allows you to see the whole engine at once.
57:58Um, allows you to see all the processes that are running, all the work that's going on on the GPU.
58:02Um, and pick out issues like, oh, this GPU is running uh like everything's slower than the others.
58:08turned out to be a NUMA awareness problem.
58:11Um, okay.
58:13So, uh, closing out here, um, if you want to learn more, mini SGA and Nano VLM are a really great place to start.
58:21These are these simple implementations of the engine architecture, um, by the two teams.
58:25They're great for humans.
58:26Um, they're incredible for agents.
58:28They can fit all the context in, you know, in their context window.
58:32Uh, Alexa Gordic did a great walkthrough blog of VLM.
58:35uh and uh share these slides later.
58:38Um also work through that in a notebook so you can play with it and then actually uh you can point your coding agent at the coding base and ask questions.
58:46Cognition al also operates this thing called deep wiki which basically crawls these a bunch of repos including sing and vlm and writes nice architecture diagrams and makes this like wiki style
58:58um page where you can sort of read and ask and then ask their um agent questions which is a nice thing to pair with your own agent setup.
59:08Um so that's everything.
59:10I'm out of time.
59:11So I'll just quickly say if you're interested in deploying and owning your own inference, we we have a a new product on modal to help you like get started with deployment really easily.
59:21Get optim get an initial baseline of reasonably optimized uh inference engine deployment that you can work off of with modal endpoints.
59:29Um so check that out and if you're interested in working on deploying inference for hundreds of of teams, uh check out modal.j jobs.
59:38Thank you.
59:40[applause]
59:54[music]