WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=g4SsanB0gMc

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/g4SsanB0gMc.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:12.480 --> 00:00:15.440
All right, I think you guys can hear me.

s2
00:00:15.440 --> 00:00:18.080
I can certainly hear myself.

s3
00:00:18.080 --> 00:00:21.760
Um, whoever came closer gets a t-shirt.

s4
00:00:21.760 --> 00:00:22.640
I I meant it.

s5
00:00:22.640 --> 00:00:25.199
There's like back full of t-shirts over here.

s6
00:00:25.199 --> 00:00:26.560
Uh, and also for questions.

s7
00:00:26.560 --> 00:00:28.640
Maybe there will be some questions at the end.

s8
00:00:28.640 --> 00:00:31.760
If you ask a question, you get the t-shirt as well.

s9
00:00:31.760 --> 00:00:39.520
And if you can guess what is on the background of this slide, you get a uh you get the t-shirt as well.

s10
00:00:39.520 --> 00:00:41.760
Any guesses?

s11
00:00:42.960 --> 00:00:44.399
What does that visualize?

s12
00:00:44.399 --> 00:00:47.440
This picture in the background?

s13
00:00:49.200 --> 00:00:53.920
No. Anybody has seen a transformer model?

s14
00:00:55.120 --> 00:00:57.520
Uh yeah, positional encoding.

s15
00:00:57.520 --> 00:00:58.719
Very good.

s16
00:00:58.719 --> 00:01:00.000
You get the t-shirt, sir.

s17
00:01:00.000 --> 00:01:00.960
All right.

s18
00:01:00.960 --> 00:01:15.920
So, today we'll discuss u basically small open source models and how they are pretty good now and how they create unique challenges when you want to serve a bunch of them in your own cloud.

s19
00:01:15.920 --> 00:01:19.280
Uh everything we'll discuss is kind of open source.

s20
00:01:19.280 --> 00:01:21.600
Uh do it yourself.

s21
00:01:21.600 --> 00:01:26.960
This is the kind of stuff you can just uh you know run a command and own the stack.

s22
00:01:26.960 --> 00:01:31.680
So there is no proprietary uh you know pieces of the puzzle here.

s23
00:01:31.680 --> 00:01:35.119
Uh let's get this uh underway.

s24
00:01:35.119 --> 00:01:36.720
Well, this works.

s25
00:01:36.720 --> 00:01:37.280
Okay.

s26
00:01:37.280 --> 00:01:38.799
So small models.

s27
00:01:38.799 --> 00:01:40.560
What do we mean by small models?

s28
00:01:40.560 --> 00:01:50.320
Um you know depending who you ask, the way I think about it is basically models that you can run on two three generations old Nvidia hardware.

s29
00:01:50.320 --> 00:01:53.759
The whole model fits into one GPU.

s30
00:01:53.759 --> 00:02:01.360
um and therefore they are easy to serve those GPUs are available and they are affordable as well.

s31
00:02:01.360 --> 00:02:15.360
Um and then most people think okay small models um there will be some kind of tradeoff in terms of um you know quality of the results and hopefully I'll be able to do a good job in this talk to convince you that

s32
00:02:15.360 --> 00:02:26.959
actually for specific tasks you can be at frontier or beyond frontier performance and get all the other obvious benefits right uh orders of magnitudes of uh kind of cost savings

s33
00:02:26.959 --> 00:02:33.760
and uh potentially quite big uh latency or throughput improvements of course.

s34
00:02:33.760 --> 00:02:39.120
Um so this is kind of one of the charts we like to show.

s35
00:02:39.120 --> 00:02:43.519
This is the artificial analysis intelligence index over time.

s36
00:02:43.519 --> 00:02:52.480
And what they typically don't show you is that there is like a breakdown of the open source models you should think about right there is the GLM 5.2 and so on

s37
00:02:52.480 --> 00:02:58.239
those kind of frontier open source models with let's say 750 billion parameters.

s38
00:02:58.239 --> 00:03:04.560
Um but then there are the small open source models kind of trailing the big ones and trailing the frontier.

s39
00:03:04.560 --> 00:03:10.400
You can see the frontier is kind of getting diminishing returns these days and the small models are catching up right.

s40
00:03:10.400 --> 00:03:21.599
So uh you see this kind of convergence saturation on top and and kind of growth of the of the small models um and you know let's say quen 3627b

s41
00:03:21.599 --> 00:03:34.000
somewhere around the performance of GPT 5.1. So if you have a workflow, if you have a pipeline that uh you know can run with GPT 5.1, now you can move that to a small model and uh

s42
00:03:34.000 --> 00:03:36.560
you know get all the all the benefits we discussed.

s43
00:03:36.560 --> 00:03:39.760
So small models not dumb anymore.

s44
00:03:39.760 --> 00:03:45.040
Um now it is also about how you use the small models, right?

s45
00:03:45.040 --> 00:03:53.760
So you can't just treat that 27 billion parameter quen 36 as your kind of totally generalized I can prompt it to do anything kind of model.

s46
00:03:53.760 --> 00:04:06.319
Now you need to adopt the approach where um you basically figure out slice of tasks from the generalized model workload and then per task you figure out which model in the open source fits the task the best.

s47
00:04:06.319 --> 00:04:16.320
You run some evals maybe some adaptation we'll discuss um and then you know that's how you kind of reach the the right quality uh to actually push this into production.

s48
00:04:16.320 --> 00:04:21.440
So here is some example of a contract review agent that uses you know nine different models.

s49
00:04:21.440 --> 00:04:28.800
This is the kind of shape that you will see in your workloads in your agents as you move to using small models for your um setup.

s50
00:04:28.800 --> 00:04:43.600
you'll start to see that okay instead of kind of hammering one API with bunch of different requests uh or one model you rather use a fleet of models and then your problem is okay how do I serve all of these different things in a way that

s51
00:04:43.600 --> 00:04:57.759
my infra people don't go crazy right and this is just one of the agents that you might be running and there might be you know 10 of these in your company so how do how do we sort of you know that's the kind of expansion of infrastructure scope let's say

s52
00:04:57.759 --> 00:05:11.759
um Now all of those different tasks that I mentioned there is an open source model that's uh sitting there waiting to be used um from you know OCR to question answering on top of documents to labeling images

s53
00:05:11.759 --> 00:05:23.120
generating SQL you know reviewing code uh there are open source models fine-tuned and trained for those tasks uh you know if you use an open source model that's trained to do OCR

s54
00:05:23.120 --> 00:05:28.880
on receipts in Vietnamese that project has seen the most receipts in Vietnamese, right?

s55
00:05:28.880 --> 00:05:36.479
There's somebody who like took the time to gather as much data as possible and on that task that model will outperform pretty much anything else.

s56
00:05:36.479 --> 00:05:42.400
Um, and there is, you know, hundreds of thousands of models on hagging phase that look like that, right?

s57
00:05:42.400 --> 00:05:47.600
So, it's just uh it's all sitting there and it's all free basically mostly quite permissive licenses.

s58
00:05:47.600 --> 00:06:02.880
So the models exist you know that that's not the bottleneck and you know we've been talking about like open source AI since 2024 and uh it's so far still not really happening and to the extent it's happening in companies

s59
00:06:02.880 --> 00:06:16.479
it basically equals like open source AI equals AWS bedrock uh except when you look at the model catalog in bedrock it's like very you know restrained in model types that are available these models are old

s60
00:06:16.479 --> 00:06:20.400
uh often you two, three years behind the state-of-the-art.

s61
00:06:20.400 --> 00:06:26.160
Um, and when you do any kind of fine-tuning in bedrock, you don't actually own the fine-tuned or trained artifacts.

s62
00:06:26.160 --> 00:06:29.759
So, you can't, you know, use it as an actual advantage in your business.

s63
00:06:29.759 --> 00:06:33.360
It kind of stays serving from the bedrock infra.

s64
00:06:33.360 --> 00:06:35.120
Um, so that's on the proprietary.

s65
00:06:35.120 --> 00:06:49.120
Now if you do small models serving on open source infrastructure blm sglank different solutions just know that these things are not tuned for any specific model or any specific hardware model combination

s66
00:06:49.120 --> 00:07:00.160
you'll have to do the tuning right this is the do-it-yourself all of these tools ship with guides on how to actually do the tuning the parameter sweep tailoring to your traffic

s67
00:07:00.160 --> 00:07:00.880
and so on.

s68
00:07:00.880 --> 00:07:06.160
This is a kind of open-ended research project every time you try to adopt one of these tools.

s69
00:07:06.160 --> 00:07:14.800
So this is not really something that sort of you you take it and it's like an engineering project and a week later you have a high performance surveying infrastructure

s70
00:07:14.800 --> 00:07:16.240
doesn't work like that.

s71
00:07:16.240 --> 00:07:19.759
Um and that's kind of the typical problem with open source tools, right?

s72
00:07:19.759 --> 00:07:22.800
This kind of like a little bit too much do it yourself.

s73
00:07:22.800 --> 00:07:35.919
Um, and then on top of this not being kind of pre-tuned for small models, the small model workloads and traffic that uses a bunch of different models kind of flips the equation for inference

s74
00:07:35.919 --> 00:07:37.680
uh kind of clusters, right?

s75
00:07:37.680 --> 00:07:45.280
So normally when you try to serve one big model, um, your problems are how do I share that model across multiple GPUs?

s76
00:07:45.280 --> 00:07:50.319
How do I have a router sitting on top that understands the state of all these workers?

s77
00:07:50.319 --> 00:07:59.039
you know the KV cache state and so on and then makes a top- down routing decision of okay this request goes to this worker or this group of workers

s78
00:07:59.039 --> 00:08:10.080
and and so on right it's very top down setup but if you have small and fast requests and you have many of them the this this sort of top down routing becomes the bottleneck

s79
00:08:10.080 --> 00:08:19.919
right because the router has a little bit obsolete version of the worker state and it's just really hard to saturate the workers if you have that kind of upfront decision on

s80
00:08:19.919 --> 00:08:37.680
that's that has to get it perfectly right in terms of u you know balancing the local cues on each of these workers because there's many small requests right um and you know like we have experimented with the VLM and SGline routers for small models and this sort of traffic

s81
00:08:37.680 --> 00:08:51.279
and it's very hard to get your GPU utilization beyond 20 30% under constant load and the problem is that those batches are just not correctly sized basically because you that routing bottleneck.

s82
00:08:51.279 --> 00:08:58.480
Um, and then the third problem is that with small models, you benefit a lot from Loras and just model adaptation in general.

s83
00:08:58.480 --> 00:09:05.600
And so the traffic that you have to serve, you know, contains, you know, people coming to you and saying, "Hey, I have 10 loras.

s84
00:09:05.600 --> 00:09:09.120
How do I, you know, use this with our serving stack?

s85
00:09:09.120 --> 00:09:12.480
Um, or I have this custom fine tune I made last night.

s86
00:09:12.480 --> 00:09:15.839
You know, um, I I want to serve this in production."

s87
00:09:15.839 --> 00:09:29.200
And this conversation between the AI engineer and the infrastructure person in getting those you know lauras up there custom models up there that's the thing that takes time and basically that's like the main killer in organizational

s88
00:09:29.200 --> 00:09:42.000
uh velocity is talking right like ideally you would want the infrastructure engineers to do their job and you would want to those AI engineers to do their job and they don't have to talk to operate on the day-to-day mode.

s89
00:09:42.000 --> 00:09:45.839
So, so you know they they are not blocking each other basically.

s90
00:09:45.839 --> 00:10:02.720
Um and this kind of model adaptation desire around small models kind of breaks that and creates a lot of back and forth and uh that's a problem right uh so these are some challenges related to okay we have a bunch of small models how do we have a cluster how do we serve this efficiently

s91
00:10:03.200 --> 00:10:21.920
um so we have been playing with this problem for a while I'm Daniel actually from superl I kind of skip the intro um so we are you know VC backed company out of SF and we have been building AI powered search and document processing systems and and agents for the last couple of years and our main painoint

s92
00:10:21.920 --> 00:10:23.839
has always been inference.

s93
00:10:23.839 --> 00:10:26.800
um specifically these problems that I have described.

s94
00:10:26.800 --> 00:10:42.800
And so we have iterated and iterated and explored different topologies for clusters for uh running you know large wide fleets of small models in different environments because sometimes you need to deploy together with some platform in some environment

s95
00:10:42.800 --> 00:10:45.279
where who knows what is available there.

s96
00:10:45.279 --> 00:10:53.440
um you know the small models make it easier because in whatever environment you can get some L4s or some kind of small GPU quota is much easier.

s97
00:10:53.440 --> 00:11:03.760
So this is kind of I'll describe a little bit about the topology of the cluster that we have kind of converged to and by the way this whole thing is Apache 2.0 So completely open source.

s98
00:11:03.760 --> 00:11:09.600
Um you guys can just take it and wrap it and now we are an inference startup.

s99
00:11:09.600 --> 00:11:16.000
Um this is open source from kind of the control plane all the way down to the thing that runs on the GPU.

s100
00:11:16.000 --> 00:11:19.760
Um so we we didn't pull any any punches.

s101
00:11:19.760 --> 00:11:31.839
Uh and the topology is basically there is a gateway and instead of having a router that kind of predecides what goes where there is a gateway that parses some of the request and attaches some metadata to the request

s102
00:11:31.839 --> 00:11:36.160
inserts that request into a shared queue and into some side channels.

s103
00:11:36.160 --> 00:11:44.800
I'll go a little bit into that and then the workers pull from that centralized queue instead of kind of pushing the data down to the workers.

s104
00:11:44.800 --> 00:11:47.920
And this way they can saturate themselves better.

s105
00:11:47.920 --> 00:11:59.440
And then the worker um setup I think I have a slide for that uh will describe how uh we basically absorb the complexity of different model architectures into kind of a coherent

s106
00:11:59.440 --> 00:12:04.480
set of workers that you know don't have like competing Python requirements and stuff like that.

s107
00:12:04.480 --> 00:12:08.079
Um so that's kind of the overall topology.

s108
00:12:08.320 --> 00:12:12.240
Um and this is kind of life of a request.

s109
00:12:12.240 --> 00:12:16.320
So maybe just uh I'll call out a couple of things from here.

s110
00:12:16.320 --> 00:12:28.880
Um we you know one of the things we don't like about the OpenAI kind of API standard is the B 64 encoded kind of JSON not good for small models not good for high throughput.

s111
00:12:28.880 --> 00:12:32.399
So we use message pack throughout like a binary format.

s112
00:12:32.399 --> 00:12:38.560
Um this way we can also push all the multimodel data through the actual API gateway.

s113
00:12:38.560 --> 00:12:49.760
So there is no like hey you know binary data over here and then request over here and then the cluster needs access to your cloud storage to start loading some b some binary data images or videos.

s114
00:12:49.760 --> 00:12:53.360
We kind of encode it all and we push it through the gateway.

s115
00:12:53.360 --> 00:13:05.760
Um and then the gateway kind of separates some of these heavier pieces to not clog the internal queue and differs it on cloud storage kind of inflight while the request is in Q. So it kind of splits up some of these requests

s116
00:13:05.760 --> 00:13:10.079
that are let's say over a megabyte and then uses cloud storage in the back end.

s117
00:13:10.079 --> 00:13:18.240
Um but as a user you push all your bits and byes into the API layer and it's kind of clean interface because of that.

s118
00:13:18.240 --> 00:13:21.839
Um basically the whole stack is rest.

s119
00:13:21.839 --> 00:13:36.800
So gateway rest the the worker is rest and then over a socket locally it kind of attaches to different runtimes and we have basically pytorch kendle and sglank on the as as a runtime

s120
00:13:36.800 --> 00:13:48.160
um and and then when we do the optimization I'll kind of go into that on how we make sure that whichever runtime we are using and whichever code is running in that runtime is the most efficient one.

s121
00:13:48.160 --> 00:13:51.440
we have an auto research loop for that basically.

s122
00:13:51.440 --> 00:13:55.600
Um but yeah, so so life of a request kind of looks like that.

s123
00:13:55.600 --> 00:14:06.160
And like one tidbit is that you really want to make sure that the gateway that's kind of the first thing that's hit by the request doesn't do too much work

s124
00:14:06.160 --> 00:14:07.920
because then it becomes a bottleneck, right?

s125
00:14:07.920 --> 00:14:10.320
So you don't even want to parse the whole request.

s126
00:14:10.320 --> 00:14:16.240
You want to be able to kind of look at the packets and uh figure out the general shape of what's coming.

s127
00:14:16.240 --> 00:14:29.279
do the annotation and then you have the workers however many workers you have hundreds of GPUs that that uh look at the queue state and then pull from there and and the queue we use u nats jetstream

s128
00:14:29.279 --> 00:14:45.920
and that thing can do you know million requests per second like that's very hard for that to become a bottleneck um so yeah like ideally you don't want to serialize this serialize as you go through all of these different components that's basically the kind of obvious thing.

s129
00:14:45.920 --> 00:14:51.519
Um this is a little animation that shows the idea behind the centralized queuing, right?

s130
00:14:51.519 --> 00:15:00.480
So instead of the top down router trying to uh you know fill in the local cues just right, which is basically impossible.

s131
00:15:00.480 --> 00:15:13.199
um you know the the whole idea is hey can we somehow centralize the queueing and can the workers uh rather pick up the task of forming their own batches with their own prediction of the cost of the batch

s132
00:15:13.199 --> 00:15:16.320
uh and and then you know become much more efficient.

s133
00:15:16.320 --> 00:15:29.279
Now one tidbit and kind of side note once you kind of start working on these things uh you realize that it's actually really hard to predict u how many things to pick up from that share queue for the batch to be really

s134
00:15:29.279 --> 00:15:38.720
like really the optimal size and so you would want some mechanism that sort of allows you to put some things back into the queue if you figure out oh like I pulled a little bit too much

s135
00:15:38.720 --> 00:15:47.120
and that's a network hop right so that's a problem and we have special optimization for that for machines that have multiple GPUs locally, right?

s136
00:15:47.120 --> 00:16:02.800
So there is a there is additional kind of machine local queueing element that takes advantage of the fact that the local processes that run on the multiple GPUs on one machine can kind of negotiate with the queue a little bit back and forth which over the network you know there's like milliseconds

s137
00:16:02.800 --> 00:16:16.959
extra that that would add and so uh we we don't do it over the network only when we collocate the um workers on multiGPU machines and you know the I mean we are not talking about like 5% differences here Right?

s138
00:16:16.959 --> 00:16:20.240
So like you centralize the queue and now you get double the throughput of the cluster.

s139
00:16:20.240 --> 00:16:23.279
So this is this is uh significant.

s140
00:16:23.279 --> 00:16:26.000
Uh I mentioned three different runtimes.

s141
00:16:26.000 --> 00:16:38.560
So basically it's either you know we write let's say for models that are encoder only we write the the PyTorch code and we kind of optimize it and we have a auto research loop that optimizes it.

s142
00:16:38.560 --> 00:16:39.519
Same for Kendall.

s143
00:16:39.519 --> 00:16:42.560
We started to play with Kendall not too long ago.

s144
00:16:42.560 --> 00:16:46.000
We still can't get it to perform anywhere near the PyTorch performance.

s145
00:16:46.000 --> 00:16:48.000
So it's a little bit more of a research project.

s146
00:16:48.000 --> 00:17:03.759
It's just the dependency like you know the worker Docker image with PyTorch is like 12 GB and the worker do basically binary statically linked binary with Kendall is maybe like 10% of that right and if you care about

s147
00:17:03.759 --> 00:17:14.160
uh kind of waking up from the cold state and loading these images on bunch of different machines the you know going from 12 gigs to a gigabyte or something like this makes makes a huge difference.

s148
00:17:14.160 --> 00:17:16.559
So that's kind of the motivation behind Kendall.

s149
00:17:16.559 --> 00:17:21.120
It's just the the getting the same performances from PyTorch is is is really hard.

s150
00:17:21.120 --> 00:17:34.960
And then SG lang we have there as a kind of uh go to baseline like we should perform as at least as well as as SGANG with the optimal tuning of all of those parameters that I mentioned that you have to do the tuning.

s151
00:17:34.960 --> 00:17:37.600
Um here is some numbers.

s152
00:17:37.600 --> 00:17:49.280
So for example when we wrap SG lang with the socket and with our kind of RAS side car um actually we can improve on the bare sglank performance just because we kind of

s153
00:17:49.280 --> 00:18:02.799
uh do something on the batching side that natively sglang doesn't do and probably you can make it to do that if you do like if you develop custom plugins into sglang and stuff like that like probably you can match

s154
00:18:02.799 --> 00:18:12.640
our performance because you know you can just push the same logic into the SG Glang core server uh but now you are developing custom code that only works with SGLAN

s155
00:18:12.640 --> 00:18:27.600
and the whole lesson here from small models is that the runtimes are super diverse right you don't want to necessarily get staged with any one particular runtime because there is you know we have I think on the order of 50 different adapters now that that we parameterize

s156
00:18:27.600 --> 00:18:40.160
for the different models and so you need to somehow deal with this kind of underlying complexity and it's probably not by building a bunch of plugins for one specific runtime it's probably some kind of abstraction

s157
00:18:40.160 --> 00:18:53.360
uh which in our case is this rust site car concept and then the socket um now I'll talk about couple different numbers but in terms of like language around benchmarking

s158
00:18:53.360 --> 00:19:05.520
you know the knee is this concept of like when you ramp up traffic on a server uh when you sort of request more and more throughput from it and it gives you more and more throughput that's when you go kind of linearly

s159
00:19:05.520 --> 00:19:10.559
up and And then at some point you hit this point where you kind of ask for more and more is not coming.

s160
00:19:10.559 --> 00:19:13.200
So you kind of flatten out and the latency goes up.

s161
00:19:13.200 --> 00:19:17.919
So we call that the the knee and it's it's like a useful concept in in benchmarking.

s162
00:19:17.919 --> 00:19:21.120
Uh because that's kind of the point of saturation, right?

s163
00:19:21.120 --> 00:19:25.120
That's that's kind of the maximal performance without hurting latency.

s164
00:19:25.120 --> 00:19:32.799
Um so just to give you some ideas of what is possible on relatively small hardware, right?

s165
00:19:32.799 --> 00:19:35.360
And different types of small models.

s166
00:19:35.360 --> 00:19:38.400
So this is measured on the RTX Pro 6000.

s167
00:19:38.400 --> 00:19:47.039
We we kind of work with Nvidia L4, you know, A100's, RTX Pro 6000, H100, that sort of range.

s168
00:19:47.039 --> 00:19:52.640
Um, again, those GPUs are much more readily available kind of on demand in any cloud.

s169
00:19:52.640 --> 00:19:55.919
Basically, most continents have quota, you know.

s170
00:19:55.919 --> 00:20:08.400
Um and on this kind of stuff uh you can basically get uh for embedding models even up to let's say uh hundreds of millions of parameters you can get hundreds of thousands

s171
00:20:08.400 --> 00:20:23.840
of tokens per second encoded into the embedding right so imagine you are sitting there now like hitting your text embedding tree on open AAI API instead you could be like having one GPU and push half a million tokens per second

s172
00:20:23.840 --> 00:20:42.880
into that thing and get the vectors out right like is this like connecting right you have half a million tokens that you are pushing into single GPU that's not even that big per second and you are getting out vector embeddings for your search system as opposed to like

s173
00:20:42.880 --> 00:20:55.360
pushing all of that into a managed embeddings endpoint somewhere and paying like orders of magnitude more money right and you can get latencies like you know low tens of milliseconds for these calls

s174
00:20:55.360 --> 00:21:06.400
like if you use uh you know cohhere openai APIs and so on these are hundreds of milliseconds right and and this is not rocket science you know you can have just like massive

s175
00:21:06.400 --> 00:21:24.159
costsaving massive latency improvements and relatively e easy operation um with with like handful of GPUs and some some infra around them right so this is like really low hanging fruit if you start anywhere with open source models small models

s176
00:21:24.159 --> 00:21:26.799
embeddings are like no-brainer, right?

s177
00:21:26.799 --> 00:21:28.559
Uh but it doesn't end there.

s178
00:21:28.559 --> 00:21:42.640
So let's say u you want to look at uh named entity recognition, you want to look at let's say multi vector search u even generation rate uh of text or structured outputs and so on.

s179
00:21:42.640 --> 00:21:59.919
um you you can be getting you know thousands of tokens per second output from uh you know task specific generative models as well per uh like let's say half a thousand per second for for one GPU there at the bottom

s180
00:21:59.919 --> 00:22:12.000
um and so let's say you are generating synthetic data you are generating annotations for your finetuning for your evals you know don't do that on a on a managed endpoint

s181
00:22:12.000 --> 00:22:14.960
That's a perfect task because you have it kind of under control.

s182
00:22:14.960 --> 00:22:16.720
You can survey the quality.

s183
00:22:16.720 --> 00:22:22.159
That's a perfect task for uh open source model on your own infra.

s184
00:22:22.960 --> 00:22:31.679
Um and then you like if the infra you have around those GPUs is like reasonable you'll get linear scaling with with the number of those GPUs.

s185
00:22:31.679 --> 00:22:45.679
Um now another sort of u idea if you are into small model serving uh is that you don't you know normally um you have kind of worker pool per model right you have a set of

s186
00:22:45.679 --> 00:22:56.480
workers set of nodes uh they have GPUs you kind of bring those up you preload the models the models load for tens of minutes because there are hundreds of billions of parameters

s187
00:22:56.480 --> 00:23:03.200
uh and so you're happy okay they finally loaded now I have a worker pool this mentality doesn't really work with small models.

s188
00:23:03.200 --> 00:23:03.520
Yeah.

s189
00:23:03.520 --> 00:23:03.840
Yeah.

s190
00:23:03.840 --> 00:23:04.480
Quickly.

s191
00:23:04.480 --> 00:23:07.200
How how what's the time left?

s192
00:23:07.200 --> 00:23:07.679
Over.

s193
00:23:07.679 --> 00:23:08.880
Oh, six minutes is over.

s194
00:23:08.880 --> 00:23:09.600
Okay.

s195
00:23:09.600 --> 00:23:09.919
All right.

s196
00:23:09.919 --> 00:23:14.000
So, pack models on the same GPU is faster.

s197
00:23:14.000 --> 00:23:27.039
Um, this is a story of how you still want to pin some models, but you want to also do u basically uh lazy loading and eviction uh as a kind of function of memory pressure.

s198
00:23:27.039 --> 00:23:29.679
You want to figure out how to combine the two.

s199
00:23:29.679 --> 00:23:33.440
Um there is a little bit about kind of auto research.

s200
00:23:33.440 --> 00:23:39.600
We have auto research loops for uh adding support for new models and for their performance.

s201
00:23:39.600 --> 00:23:47.360
Um we build a lot of internal tooling to do the measurement to feed into those out research loops to basically push the numbers forward.

s202
00:23:47.360 --> 00:23:53.600
Um and maybe perhaps most importantly when we ship support for a model it has all the tuning done right.

s203
00:23:53.600 --> 00:23:56.159
So there is no okay let's do a parameter sweep.

s204
00:23:56.159 --> 00:24:00.240
We bundle basically a config for end to end the whole cluster.

s205
00:24:00.240 --> 00:24:04.159
Um this is a setup for the auto research loop.

s206
00:24:04.159 --> 00:24:12.720
There is like a meta loop that builds the harness that then runs the loop and there is a dashboard on top that helps you understand how it works.

s207
00:24:12.720 --> 00:24:16.000
Um, we have custom UIs for that.

s208
00:24:16.000 --> 00:24:28.400
And one of the outputs of that was a Laura that took 80 cents to train and it improved 18 it improved quality of retrieval on German legalist STEX as a proof of concept by 18%.

s209
00:24:28.400 --> 00:24:29.600
And that's it.

s210
00:24:29.600 --> 00:24:31.520
So small models are good.

s211
00:24:31.520 --> 00:24:33.760
They are relatively easy to serve.

s212
00:24:33.760 --> 00:24:44.000
uh they are actually much cheaper, faster, is as smart and that QR code goes to the GitHub repo of our cluster that I just described.

s213
00:24:44.000 --> 00:24:46.960
Give us a star and happy self-hosting.

s214
00:24:46.960 --> 00:24:48.832
Thank you.

s215
00:24:48.832 --> 00:24:50.832
[applause]