WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=_cRgl3maiT8

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/_cRgl3maiT8.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.480 --> 00:00:13.200
Hello everyone.

s3
00:00:13.200 --> 00:00:14.920
My name is Daniel.

s4
00:00:14.920 --> 00:00:18.880
I work on AI and AI agents here at Cerebras.

s5
00:00:18.880 --> 00:00:19.880
What's your name?

s6
00:00:19.880 --> 00:00:20.600
I'm Michelle.

s7
00:00:20.600 --> 00:00:28.280
I'm a co-founder at Gimlet Labs where we're building a multi-silicon neo cloud to optimize inference at every layer of the stack.

s8
00:00:28.280 --> 00:00:36.040
And we're really excited to bring you our talk, all the things we have to do to satisfy your insatiable need for tokens.

s9
00:00:36.520 --> 00:00:39.560
As you know, AI spend is going kind of bananas right now.

s10
00:00:39.560 --> 00:00:49.440
With this year we're slated to spend $765 billion on compute and in 2031 up to $1.6 trillion.

s11
00:00:49.440 --> 00:01:03.160
The frontier labs are having a mad dash to secure as much compute as possible to buy across hardware vendors to do both training for the frontier models as well as to serve all of their users.

s12
00:01:03.160 --> 00:01:09.680
And all of their compute is going to satisfy everyone's insatiable need for tokens.

s13
00:01:09.680 --> 00:01:15.800
Google at their IO last year said that they were serving 480 trillion tokens per month.

s14
00:01:15.800 --> 00:01:21.600
And last month they said that it jumped up to 3.2 quadrillion tokens per month.

s15
00:01:21.600 --> 00:01:28.920
So all of this is happening at the same time as models are getting cheaper and faster.

s16
00:01:28.920 --> 00:01:37.680
Lama 3.37 B when it was first released compared to now is now 25% cheaper whilst still being 3.6 times faster.

s17
00:01:37.680 --> 00:01:41.040
So in today's talk we're going to talk about two things.

s18
00:01:41.040 --> 00:01:46.720
One, what have we done so far to optimize inference to make it cheaper and faster.

s19
00:01:46.720 --> 00:01:52.120
And what does the road map look like to juice perf and throughput even further?

s20
00:01:52.320 --> 00:02:03.920
So to do this, let's take a little bit of a time jump to 2022 and learn about all the optimizations that led us to the point we're at now and hopefully what the future looks like.

s21
00:02:03.920 --> 00:02:10.920
So, LLMs produce tokens one at a time in something called an auto-regressive loop.

s22
00:02:10.920 --> 00:02:17.120
So, to generate one additional token, you need to read all of the weights from memory into compute.

s23
00:02:17.120 --> 00:02:19.880
So, most of the time, the computer is sitting idle.

s24
00:02:19.880 --> 00:02:25.080
It's just waiting for memory to be able to stream in all the weights necessary to do the compute itself.

s25
00:02:25.080 --> 00:02:29.640
So, up to 90% of the time, the computer is just sitting there waiting on memory to load.

s26
00:02:29.640 --> 00:02:37.959
So, how can we figure out how to optimize this so there's less idle time and you can make full advantage and serve tokens faster?

s27
00:02:37.959 --> 00:02:40.360
So, what you can do is throw more compute at the problem.

s28
00:02:40.360 --> 00:02:50.320
So, instead of streaming in the whole set of weights, what if you could split up the entire model into four and have each GPU take care of a fourth of the model each?

s29
00:02:50.320 --> 00:02:56.800
Then, you could reduce, in theory, the amount of time it takes to serve a single token by four times.

s30
00:02:56.800 --> 00:02:59.400
However, you have to pay a tax if you want to do this.

s31
00:02:59.400 --> 00:03:10.880
Before the attention block and then the F So, after the attention block and then the FFM block, you have to do something called all-reduce, where you take each of the individual calculations from each GPU

s32
00:03:10.880 --> 00:03:12.600
and you combine it all together.

s33
00:03:12.600 --> 00:03:16.800
So, you can actually get a calculation because each of the GPUs doesn't have the whole picture.

s34
00:03:16.800 --> 00:03:18.600
It only does 1/4 of the model.

s35
00:03:18.600 --> 00:03:27.800
So, you need to do this across every layer twice, which can lead to only getting 2x speed up for getting 4x the amount of compute that you add on.

s36
00:03:27.800 --> 00:03:37.040
If I were to graph this in a pretty graph, um you can see that throwing more GPUs at the problem does get you faster tokens, but it requires exponentially

s37
00:03:37.040 --> 00:03:45.000
more energy to serve a single token as you put in more compute at the problem because of the communication overhead we just talked about.

s38
00:03:45.000 --> 00:03:56.800
So, if we flip the Y axis here so it's throughput, you can see that this resembles something like a Pareto frontier, where you can see that you have to make some kind of sacrifice between throughput and speed

s39
00:03:56.800 --> 00:03:58.600
when you're an inference provider.

s40
00:03:58.600 --> 00:04:08.640
A great example of this is Claude code fast mode, where you can spend, I think, 6x the amount of tokens to serve a single token compared to regular Claude for twice the speed.

s41
00:04:08.640 --> 00:04:18.560
So, for that product, they're really optimizing towards the right of the Pareto frontier, where they're getting the speed, but at the cost of not being able to serve as many tokens.

s42
00:04:18.560 --> 00:04:27.360
So, during today's talk, we're going to talk about what are the innovations we have done in the past and in the future to be able to shift the Pareto frontier to the right.

s43
00:04:27.360 --> 00:04:34.120
So, you all already know from our previous slides that to generate each token, you have to read the whole model.

s44
00:04:34.120 --> 00:04:40.440
So, what if we could spend one pass that we already paid for to serve multiple requests?

s45
00:04:40.440 --> 00:04:45.880
So, that is where the idea of batching and continuous batching comes in.

s46
00:04:45.880 --> 00:04:53.480
The idea of batching is when you take a weight read that you've already done and be able to serve multiple user requests.

s47
00:04:53.480 --> 00:05:02.640
However, the problem, though, is that some of these requests are longer than others, which means that there's all of these idle periods in the compute where nothing is happening.

s48
00:05:02.640 --> 00:05:06.240
So, how can we solve this problem and squeeze out more performance?

s49
00:05:06.240 --> 00:05:09.160
That's where continuous batching comes in.

s50
00:05:09.160 --> 00:05:20.000
So, the idea here is that for every single idle slot, you slot in a new user request to be served during that same weight read to be able to serve as many requests as possible.

s51
00:05:20.000 --> 00:05:31.720
However, this does come with a slight downside because every single token when you do inference requires you to save something into the KV cache because that's how the attention block works.

s52
00:05:31.720 --> 00:05:37.640
You need to understand how every single token, like what the K and V vectors are.

s53
00:05:37.640 --> 00:05:39.560
This can add up very quickly.

s54
00:05:39.560 --> 00:05:50.120
So, if you are running a coding agent with very long context, for example, 128k context, you're going to save 42 gigabytes of KV cache that is needed to serve that individual request.

s55
00:05:50.120 --> 00:05:57.480
So, the rule of thumb here for a 70 billion parameter model is that for every 3,000 tokens, you need 1 GB of additional KV cache.

s56
00:05:57.480 --> 00:05:59.560
And that's for one user request.

s57
00:05:59.560 --> 00:06:09.080
So, imagine you're doing batching and you're doing like eight different requests at the same time, then your KV cache is going to explode and you will not have enough memory to be able to serve all of your requests.

s58
00:06:09.080 --> 00:06:13.600
So, what are some of the innovations we have done to be able to optimize how we use memory?

s59
00:06:13.600 --> 00:06:18.040
So, that's where paged KV cache comes into play.

s60
00:06:18.040 --> 00:06:28.720
So, how traditionally KV cache worked was that you would have physical memory that you would reserve to make sure that you could have enough room to be able to store all of the KV information

s61
00:06:28.720 --> 00:06:30.760
from each request into memory.

s62
00:06:30.760 --> 00:06:39.280
However, this came with a downside and a lot of inefficiency because you would just have a lot of memory blocks that would be reserved but wouldn't actually be used.

s63
00:06:39.280 --> 00:06:51.760
So, the idea with a paged KV cache is we use something that's similar to like an OS or a virtual machine where you provision a whole memory block that's all virtual and you can store wherever there's free space

s64
00:06:51.760 --> 00:06:53.680
KV cache for each request.

s65
00:06:53.680 --> 00:07:00.880
So, this allows you to serve three times more requests than not having a paged KV cache and just doing what was before.

s66
00:07:00.880 --> 00:07:08.440
And this really reduces the amount of wasted space where you are reserving memory slots without actually using them.

s67
00:07:08.440 --> 00:07:13.040
This was right after followed by an innovation called prompt caching.

s68
00:07:13.040 --> 00:07:22.040
So, the idea here is that if you have a coding agent or if you have a chat app that has a very long system prompt, you don't actually have to recalculate

s69
00:07:22.040 --> 00:07:25.080
the KV cache needed every single time you have a request.

s70
00:07:25.080 --> 00:07:31.320
What you can do is you can reuse the KV cache that you calculated already for the previous request for the next request.

s71
00:07:31.320 --> 00:07:34.080
That way you can just pick up where you left off.

s72
00:07:34.080 --> 00:07:39.800
Here, you can skip the very, very time-consuming prefill phase.

s73
00:07:39.800 --> 00:07:42.960
And you might be wondering, what is prefill?

s74
00:07:42.960 --> 00:07:48.760
So, during inference, you can split up what happens into two major phases, prefill and decode.

s75
00:07:48.760 --> 00:07:57.160
So, the prefill phase consists of all the things that happen once the request arrives, and then ends when the first tokens gets generated.

s76
00:07:57.160 --> 00:08:05.480
And here, you can see this when you type something into ChatGPT, and then it takes a couple seconds before it starts returning the first token, and then it goes kind of like

s77
00:08:05.480 --> 00:08:06.600
smoothly from there.

s78
00:08:06.600 --> 00:08:15.240
So, what's happening during the first couple of seconds where it's not returning anything is that it's computing K and V vectors for every single prompt token that you've inputted

s79
00:08:15.240 --> 00:08:17.120
and produces the first output token.

s80
00:08:17.120 --> 00:08:20.480
This is very compute bound because you can optimize the hell out of this.

s81
00:08:20.480 --> 00:08:23.880
You can parallelize this extremely.

s82
00:08:24.000 --> 00:08:33.400
However, after the first token is generated, what happens is uh that you decode one token at a time, reusing the KV cache you just built in the prefill phase.

s83
00:08:33.400 --> 00:08:36.240
So, this is more reliant on memory.

s84
00:08:36.240 --> 00:08:46.000
The reason I'm talking about this is because something happened lately that's really messing up a lot of inference pipelines, which is super super token hungry coding agents.

s85
00:08:46.000 --> 00:08:51.839
So, let's say you have a coding agent that's using 200K context or 500K context or a million context, right?

s86
00:08:51.839 --> 00:08:54.960
What happens is that it starts to clock up the pipeline.

s87
00:08:54.960 --> 00:08:59.280
While prefill is running, no other prefill or decode request can get through.

s88
00:08:59.280 --> 00:09:06.880
So, this is kind of catastrophic if you have a bunch of people that are sending in coding requests and a couple people that are just trying to figure out like math

s89
00:09:06.880 --> 00:09:09.400
problems in chat mode.

s90
00:09:09.400 --> 00:09:17.760
So, to avoid this clogging problem, there is an innovation called chunked prefill that kind of goes and solves this problem.

s91
00:09:17.760 --> 00:09:20.000
And the idea is quite simple here.

s92
00:09:20.000 --> 00:09:28.600
The idea is taking a very long piece of prefill, chopping it up into smaller bits, so you can interleave in different decode and prefill requests.

s93
00:09:28.600 --> 00:09:33.880
That way, there's not one thing that's taking up a bunch of time and clogging the whole pipeline.

s94
00:09:33.880 --> 00:09:38.200
So, now Michelle, tell us more about what else we can optimize.

s95
00:09:38.200 --> 00:09:43.160
Yeah, so all of the optimizations that Daniel just talked about are run time level optimizations.

s96
00:09:43.160 --> 00:09:49.320
So, that explains we explore how we make inference more efficient within a single worker.

s97
00:09:49.320 --> 00:09:59.960
But, in order to to keep pushing that frontier, we really need to start looking at system level optimizations, which is how can we distribute these workloads across many different workers.

s98
00:10:00.600 --> 00:10:04.240
And so, the first thing to recognize there is that every request is different.

s99
00:10:04.240 --> 00:10:07.040
So, take for example a summarization use case.

s100
00:10:07.040 --> 00:10:13.240
You're throwing the LLM a bunch of different documents, and you just really want a single question answered.

s101
00:10:13.240 --> 00:10:17.400
And so, this workload is going to have a long prompt and short answer.

s102
00:10:17.400 --> 00:10:22.080
And so, it's a dominated by that prefill phase, and so therefore compute bound.

s103
00:10:22.080 --> 00:10:25.480
On the other hand, you might have a case where it's the opposite.

s104
00:10:25.480 --> 00:10:31.520
Let's say you want to write a long story or book, you just want to pass it a few themes for it to go off of.

s105
00:10:31.520 --> 00:10:34.840
And so, this is going to have a short prompt and long answer.

s106
00:10:34.840 --> 00:10:41.000
And this is going to be dominated by decode because you have so many output tokens compared to the number of input tokens coming in.

s107
00:10:41.000 --> 00:10:43.080
And this is going to be memory bound.

s108
00:10:43.080 --> 00:10:46.320
And finally, you can have a case that's somewhere in the middle, right?

s109
00:10:46.320 --> 00:10:54.800
It's both a long prompt, long answer, and you're seeing here that both the prefill and decode phases and therefore compute and memory both matter.

s110
00:10:54.800 --> 00:11:04.640
And by looking at these different examples, you can really see that the resource requirements that you have for your hardware, it's pretty different depending on what your workload looks like.

s111
00:11:04.640 --> 00:11:10.120
And so, how do you prevent yourself from getting, you know, bound by compute, bound by memory?

s112
00:11:10.120 --> 00:11:13.320
And so, one way to do that is called disaggregation.

s113
00:11:13.320 --> 00:11:17.560
So, splitting up your workload to different worker nodes.

s114
00:11:17.560 --> 00:11:26.040
And so, the most common form is called prefill decode disagg, where you're actually splitting and disaggregating this workload by phase.

s115
00:11:26.480 --> 00:11:29.280
And what that allows you to do is essentially have two pools.

s116
00:11:29.280 --> 00:11:33.000
So, you have your prefill pool, that's going to be running the prefill part of your workload.

s117
00:11:33.000 --> 00:11:36.200
The decode pool, that's going to be running the decode part of your workload.

s118
00:11:36.200 --> 00:11:46.880
And taking the summarization use case for example, you know, you're going to want to scale up that prefill pool so you're not blocked by the compute and scale down your decode pool because

s119
00:11:46.880 --> 00:11:51.720
you don't want to waste resources when you just really don't have that much decode work to do.

s120
00:11:51.720 --> 00:11:58.800
And it's nice cuz now you can scale all these independently and neither part is stalling the other.

s121
00:11:59.080 --> 00:12:01.040
And we can take that one step further, right?

s122
00:12:01.040 --> 00:12:05.360
We said before, prefill, that workload itself is inherently bound by compute.

s123
00:12:05.360 --> 00:12:07.320
Decode, that is bound by memory.

s124
00:12:07.320 --> 00:12:17.520
And just walking through that really quickly again, the reason for that is as you're doing decode, you're stepping through each of these tokens one by one and for every decode step, you're re-reading

s125
00:12:17.520 --> 00:12:18.720
the entire KB cache, right?

s126
00:12:18.720 --> 00:12:21.560
That's going to be memory bound.

s127
00:12:21.640 --> 00:12:31.000
So yeah, um the the thing that Michelle just said about this being memory bound, this can be clearly seen by what the hardware can actually do.

s128
00:12:31.000 --> 00:12:32.680
So GPUs, right?

s129
00:12:32.680 --> 00:12:39.040
Um the the speed at which you can transfer data from memory into compute is limited by something called memory bandwidth.

s130
00:12:39.040 --> 00:12:43.320
So for an H100, this looks like 3.35 terabytes per second.

s131
00:12:43.320 --> 00:12:50.040
However, Cerebras has 7,000 times the bandwidth of a H100 with 21,000 terabytes per second.

s132
00:12:50.040 --> 00:12:56.360
This is because we have a very specialized SRAM based memory architecture on our chip.

s133
00:12:56.360 --> 00:12:59.480
So this really means that Cerebras is really built for decode.

s134
00:12:59.480 --> 00:13:08.280
So we can serve tokens at 3,000 tokens per second versus an H100 serving it at 600 tokens per second for decode.

s135
00:13:08.480 --> 00:13:12.280
So we're really excited to partner with AWS to productionize this.

s136
00:13:12.280 --> 00:13:23.960
So we are going to partner with AWS Trainium so they can serve prefill using their AWS Trainium chips, use their EFA service to transfer the KB cache to our Cerebras CS-3 systems

s137
00:13:23.960 --> 00:13:26.000
to finally serve the the token.

s138
00:13:26.000 --> 00:13:30.800
So this will lead us to give us 5x high-speed token capacity in the same footprint.

s139
00:13:30.800 --> 00:13:36.640
So, this is one of the things that we're doing to try to make fast inference more accessible and cheaper for developers.

s140
00:13:36.640 --> 00:13:39.800
So, we're really excited to really pursue this partnership.

s141
00:13:39.800 --> 00:13:44.080
And at Gimlet, we're really generalizing that beyond just training them and Cerebras.

s142
00:13:44.080 --> 00:13:55.040
What Gimlet is building is essentially a multi-silicon inference cloud, which takes your workload, breaks it down into finer-grain pieces, determines the resource characteristics of all of those different pieces,

s143
00:13:55.040 --> 00:13:59.800
and then figures out the best way to map it to the underlying hardware, where that hardware can be really anything, right?

s144
00:13:59.800 --> 00:14:03.560
Nvidia, AMD, Trainium, Cerebras, D-Matrix, Intel Gaudi.

s145
00:14:03.560 --> 00:14:05.320
The list goes on and on.

s146
00:14:05.320 --> 00:14:11.840
And by running those tasks on the machine that is most efficient for running it, we're actually seeing pretty crazy results.

s147
00:14:11.840 --> 00:14:18.400
So, we're seeing three to 10 times faster inference compared to homogeneous systems that have the same power budget.

s148
00:14:18.400 --> 00:14:29.240
And this is diving into more than just the pre-full decode disaggregation than before, and some of those techniques are, for example, disaggregating based on model instead of on phase.

s149
00:14:29.240 --> 00:14:32.120
And so, that's called speculative decoding.

s150
00:14:32.120 --> 00:14:39.760
So, just reviewing again the decode phase, typically you do a forward pass for each of the decode tokens that you're getting out.

s151
00:14:39.760 --> 00:14:41.720
And that's the standard way of doing it.

s152
00:14:41.720 --> 00:14:45.480
What speculative decode is is that you essentially have a draft model.

s153
00:14:45.480 --> 00:14:52.360
And this is just a smaller form of the full model that um can generally approximate the results that you get out of it.

s154
00:14:52.360 --> 00:14:57.480
And the nice thing there is because it's smaller, it runs a lot more quickly and also uses less resources.

s155
00:14:57.480 --> 00:15:06.800
And so, in speculative decode, what you can do is, in order to get, you know, the nine next tokens, you will essentially run the decode on that draft model first.

s156
00:15:06.800 --> 00:15:09.040
It's going to give you back what is called candidate tokens.

s157
00:15:09.040 --> 00:15:18.640
So, you're not 100% sure all these tokens are right, but because, you know, often these have been fine-tuned or been trained off of the full model, it's it's approximately right.

s158
00:15:18.640 --> 00:15:22.520
And you want to validate, okay, are these the tokens that I actually want to output?

s159
00:15:22.520 --> 00:15:30.960
And so you'll actually run the next phase on that full model that's called the verify stage to validate, okay, which of these tokens are correct?

s160
00:15:30.960 --> 00:15:35.600
And essentially you can keep doing this until, you know, you've validated all these tokens from the draft model.

s161
00:15:35.600 --> 00:15:44.600
And at the end of the day you can see here on the top, the standard way of running it, you needed to run that forward pass on your full model nine times to get those nine tokens.

s162
00:15:44.600 --> 00:15:49.520
On the other hand, with speculative decode, you actually only needed to run it three times on that full model.

s163
00:15:49.520 --> 00:15:59.440
And if you can get that draft model to run really, really quickly, then that time is minimal and you can really shorten the amount of time it takes to get your tokens.

s164
00:15:59.600 --> 00:16:09.520
And looking at some more concrete results, so this is essentially that Pareto frontier that Daniel was showing earlier where x-axis is the speed at which you're getting the tokens and the y-axis

s165
00:16:09.520 --> 00:16:10.600
is the throughput.

s166
00:16:10.600 --> 00:16:16.320
And so the baseline case we're comparing to here is prefill decode disagg across homogeneous GPUs.

s167
00:16:16.320 --> 00:16:23.760
And you can see that, you know, throughput is pretty good, but as soon as you're pushing on that speed, that throughput drops pretty drastically.

s168
00:16:23.760 --> 00:16:27.680
So what if you then run speculative decode across those same GPUs?

s169
00:16:27.680 --> 00:16:32.760
So now you're running that draft model and you're able to see you're able to get that curve to the right.

s170
00:16:32.760 --> 00:16:37.480
And the reason for that is because you're running the decode on that full model less times.

s171
00:16:37.480 --> 00:16:38.920
But we can push that even further, right?

s172
00:16:38.920 --> 00:16:45.520
Like going back to our theme of running those workloads on the hardware that can run it best.

s173
00:16:45.520 --> 00:16:53.720
On something such as, you know, D Matrix or Cerebrus where you're able to run small models very, very quickly, you see drastic performance improvements.

s174
00:16:53.720 --> 00:16:57.800
So here this is where you're seeing those numbers for that same throughput, right?

s175
00:16:57.800 --> 00:17:04.640
You're able to get 10x the interactivity compared to if you're just running it on homogeneous machines.

s176
00:17:05.199 --> 00:17:09.360
The final form of disagg we wanted to talk about is called attention FFN disagg.

s177
00:17:09.360 --> 00:17:13.000
And this one gets pretty exciting when you think about serving large models.

s178
00:17:13.000 --> 00:17:22.439
And I'm saying like 3 trillion parameter models where these models are so large that even just running one layer, it can be pretty resource-intensive.

s179
00:17:22.439 --> 00:17:24.680
And so the concept is essentially the same thing, right?

s180
00:17:24.680 --> 00:17:35.520
Like for that particular task that you need to do, whether it's that attention layer or the feedforward FFN network layer, then you are essentially trying to figure out, okay, what are the resource characteristics,

s181
00:17:35.520 --> 00:17:37.240
then what is the best hardware to map it to?

s182
00:17:37.240 --> 00:17:39.800
And so the attention layer, that's going to be more memory bound.

s183
00:17:39.800 --> 00:17:42.080
It's doing a lot of KB cache accesses.

s184
00:17:42.080 --> 00:17:45.080
The FFN layer, that's doing a lot of dense matrix multiplications.

s185
00:17:45.080 --> 00:17:46.680
And so then it makes sense, right?

s186
00:17:46.680 --> 00:17:55.360
You're going to run the attention on the machine that has memory bandwidth, the FFN layer on the machines that can run those matrix multiplications as quickly as possible.

s187
00:17:55.360 --> 00:18:08.400
And essentially what happens here is even within the model, you're ping-ponging across, you know, many different machines in order to to get the inference from that model and just really disaggregating every single layer.

s188
00:18:08.400 --> 00:18:12.800
And one thing that we really wanted to note here is that there is no best form of disagg.

s189
00:18:12.800 --> 00:18:15.200
There is actually no best hardware.

s190
00:18:15.200 --> 00:18:25.360
It really depends on the tradeoffs that you have for, you know, depending on what your model is, your workload, and what you need for latency, and what you need for throughput guarantees.

s191
00:18:26.400 --> 00:18:33.680
And as we look towards the future, we see that agents are only becoming more and more incorporated into our everyday life.

s192
00:18:33.680 --> 00:18:35.000
It's incorporated into our work.

s193
00:18:35.000 --> 00:18:36.200
Everyone is using it.

s194
00:18:36.200 --> 00:18:40.360
And it's continuing to push that need for more and faster tokens.

s195
00:18:40.360 --> 00:18:42.560
Yeah, it's really getting geared to the right.

s196
00:18:42.560 --> 00:18:42.682
I

s197
00:18:42.682 --> 00:18:42.800
[laughter]

s198
00:18:42.800 --> 00:18:46.000
told Daniel not to say that.

s199
00:18:46.000 --> 00:18:56.080
But the good thing about this is that even though, you know, this seems like a crazy crazy target, we can actually use agents to help us solve these problems and keep pushing the frontier.

s200
00:18:56.080 --> 00:19:01.680
And so one of the things that we're doing at Gimlet is actually using an agent to generate optimized kernels.

s201
00:19:01.680 --> 00:19:09.120
Especially for our use case where we want to support such a wide range of different kinds of hardware, we need to figure out how do you make that problem a lot more tractable.

s202
00:19:09.120 --> 00:19:19.400
Cuz right now, how people generate and how people optimize kernels is that there's a lot of people sitting there hand tuning and figuring out how to optimize these kernels.

s203
00:19:19.400 --> 00:19:29.360
And the thing is that these optimizations can vary pretty drastically based on you know, the model that you're trying to run, the hardware that you're running on, even within generations of the hardware,

s204
00:19:29.360 --> 00:19:31.720
and you know, the workload shape itself.

s205
00:19:31.720 --> 00:19:36.920
And so what we do at Gimlet is we built KForge, which is an agent for generating these kernels.

s206
00:19:36.920 --> 00:19:47.520
It takes in a bunch of information like your hardware profiles, your workload profiles, basically anything that you can use to figure out how to optimize kernels and we solve it as a brute force problem, right?

s207
00:19:47.520 --> 00:19:49.320
And so here's an example, right?

s208
00:19:49.320 --> 00:19:54.280
We have a forward pass and the kernel that um KForge optimized from it.

s209
00:19:54.280 --> 00:19:56.240
And we're seeing pretty interesting results.

s210
00:19:56.240 --> 00:20:04.760
So last week, you know, we were running the agent for probably less than the span of a week and within that time it was able to speed up GPT-OSS-120B

s211
00:20:04.760 --> 00:20:07.880
on AMD hardware by 33% end-to-end.

s212
00:20:07.880 --> 00:20:14.760
And the interesting thing to note here is that this is building on top of kernels that have already been optimized.

s213
00:20:15.760 --> 00:20:27.120
And you can go and take, you know, agents to the next level and use them to help us build and design custom silicon that is hyper tuned for specific models or your specific workload.

s214
00:20:27.120 --> 00:20:35.760
This can then be used to train even better and smarter and faster AI, which then can also be used then to design the next generation of that custom silicon.

s215
00:20:35.760 --> 00:20:38.800
And you know, this loop keeps going on and on and it's a recursive loop.

s216
00:20:38.800 --> 00:20:42.240
Like shout out to the folks at Recursive.

s217
00:20:42.680 --> 00:20:51.400
And I want to end the talk by saying that we're just getting started in our journey to optimize throughput and speed on inference.

s218
00:20:51.400 --> 00:20:58.240
Because at the very top of the stack, you don't think really about inference optimization when you talk about like incident response, right?

s219
00:20:58.240 --> 00:21:06.720
But when you have agents that are making our systems much more reliable and increasing uptime, we're getting more throughput and we can get more tokens.

s220
00:21:06.720 --> 00:21:09.280
Or you can apply agents to the problem of bringing up models.

s221
00:21:09.280 --> 00:21:15.680
As soon as a model drops on the market, getting it running on all the different kinds of hardware on day one.

s222
00:21:15.680 --> 00:21:18.360
You can even go a little bit crazier with compilers.

s223
00:21:18.360 --> 00:21:24.360
Imagine a world where an inference that can be custom-built for your workload and the model that you're running.

s224
00:21:24.360 --> 00:21:27.080
This is already being demonstrated by folks at Anthropic, right?

s225
00:21:27.080 --> 00:21:29.040
That built a compiler in 2 weeks.

s226
00:21:29.040 --> 00:21:39.080
So, I think there's a lot of optimizations that you can do when you're building just-in-time custom software to support the model you want to run, on the workload you want to run, on your custom hardware.

s227
00:21:39.080 --> 00:21:49.160
And referring back to the previous slide, using agents to continue to help us to build, you know, more custom silicon and helping speed up the tape-out process.

s228
00:21:49.160 --> 00:21:55.440
And finally, this is the non-very technical part, but all of the things that surround what takes to serve inference.

s229
00:21:55.440 --> 00:21:59.480
All the procurement, all the legal, all of the things that are in the back office.

s230
00:21:59.480 --> 00:22:04.040
All of these things combined with agentic workflows are able to be optimized.

s231
00:22:04.040 --> 00:22:11.440
So, when we do all of this, I'm really optimistic about the future and all the things we can accomplish to serve tokens faster and cheaper.

s232
00:22:11.440 --> 00:22:13.040
So, next slide, please.

s233
00:22:13.040 --> 00:22:15.120
Thank you so much for coming to our talk.

s234
00:22:15.120 --> 00:22:19.320
Uh we are working both very, very hard to give you faster and cheaper tokens.

s235
00:22:19.320 --> 00:22:21.000
And yeah, let us know if you have any questions.

s236
00:22:21.000 --> 00:22:22.360
We'll be around after the talk.

s237
00:22:22.360 --> 00:22:24.000
Yeah, and go eat tokens.

s238
00:22:24.000 --> 00:22:27.720
We're making more tokens for you so you can also generate GIFs like this.

s239
00:22:27.720 --> 00:22:28.005
Yeah.

s240
00:22:28.005 --> 00:22:30.005
[laughter]

s241
00:22:44.380 --> 00:22:46.380
[music]