What Makes Open Models Fast in Production — Sujee Maniyam, Nebius

AI Engineer · 20 min · 221 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12Thank you for joining us today.
00:13Um I'm Dylan.
00:14I'm here today with my colleague Suji.
00:17So I lead the product marketing for Nibbius token factory and I'm here with Suji as a developer advocate and we're going to talk a bit about engineering um open NLMs for production.
00:28So before we jump right in, um I want to give a few words about what we're going to cover today.
00:32We only have 20 minutes, so we'll try to keep this brief.
00:35Uh but we're going to give you a quick intro on who we are, what we do.
00:38Uh then talk about what it takes to build the infra for inference.
00:42Um a few words on model shaping optimizations, which are extremely important in what we do.
00:47And then we'll talk about model serving optimizations.
00:52But before we jump right in, I just want to give a few words uh about who we are and what we do.
00:56So we work for Nibbius.
00:59Uh so specifically Nibbius token factory but Nibbius is a like a full stack AI cloud infrastructure.
01:05Um so this means we actually not only exposing model APIs we run the actual physical uh AI layer and infrastructure underneath it.
01:12Um we have our own data centers and video systems uh bare metal capacity manage cloud and optimizing the serving and the inference through Nibius token factory.
01:23Um top top row here is really about scale and trust.
01:26So we're a publicly traded company on the NASDAQ um headquartered in Amsterdam.
01:31Uh we're very close with Nvidia who made a 2 billion uh investment in Nebus a couple months ago and we're targeting a 5 gawit of Nvidia systems by the end of 2030.
01:42Uh bottom row is really more about the control of the stack.
01:45This is a quite a a good edge we have.
01:47We operate own data centers across the EU and the US uh bare metal rather than shared cloud.
01:53We're a very early adopter of Nvidia Rubin um CP like uh and bluefield storage and we were amongst the first European clouds to run both Blackwell Ultra AGXP300 and GB300
02:04in production.
02:06Uh we also have um like strong contracts from Microsoft and Meta agreements that kind of show that the largest AR builders are already building this kind of capacity from us.
02:15Um and the last time is quite important.
02:18It's the uh like really the important bridge to this talk is the vertical integration from silicon to serve that we um rely on.
02:28So today we're talking about NEA token factory.
02:30So the idea here is to like really uh engineer AI for product.
02:34So we're managed inference company uh meaning we deploy these models and optimize them for our customers.
02:41So we noticed a few things today in the market of serving open source LLNs.
02:44Um, usually most AI teams are stuck between choosing two bad options.
02:49Um, closed APIs are really easy to start with, but you often hit a ceiling really fast and you can't really tune the model to your specific use case.
02:56Uh, you share the infra with everyone else.
02:59So, it kind of operates as a black box black box and the costs grow in a straight line with no way to really optimize things.
03:06The other option here is self-hosting.
03:08Obviously, it's quite fun.
03:09Uh, gives you full control on the model, but it's a massive engineering project.
03:13Um so you really need a dedicated team to actually keep it running.
03:17Um and you're usually month away from production before you even start your actual product.
03:21So token factory is kind of the shift uh this third path we identified uh before you even start like on your um actual product.
03:29You get the control and the performance of self-hosting with the simplicity of a manage inference service.
03:33So we do the heavy lifting, the hard infrastructure work and then customers stay focused on their product.
03:39This really unlocks uh three main things performance, cost and behavior of the models.
03:44Um as we've seen the gap between proprietary and open source models has been closing a lot and so this is where we kind of operate and we feel comfortable um operating is really not only
03:54relying on the fact that this bridge and this gap is closing down but also figuring out how you actually make it work in your own production uh use cases.
04:04So this is really the core of token factory.
04:06uh we've built a full stack JI platform.
04:08Um most platform we've seen on the market either give you inference training uh data tooling but here we decided to connect everything under one loop.
04:16So we have a bunch of tools to that kind of unlock this on the inference side um we allow you to run models in production.
04:23So we have 60 plus models on the platform JLM Kimmy DC you name them.
04:29Um we have dedicated endpoints we have different flavors on the models depending if you want to run them fast.
04:34uh we have different tooling like structured outputs, function calling and batch API.
04:38Second layer here is the data lab.
04:40So this allows you to really capture and structure the production logs.
04:43So we see a lot of customers actually running these models in production importing the completions and wanted a way to kind of slice and dice through these completions and this is really what we allow with the data lab.
04:52So uh inference log imports SQL data like uh data set filtering, data set versioning, um exporting and then batching.
05:00These uh third layer is post training.
05:02So we allow you to kind of fine-tune distill optimize these models.
05:06So this is a natural follow-up with the data lab is once you have your logs and your inference and you can generate synthetic data sets you can bring them all to post training and use that to kind of either through Laura or full fine tuning make the models much more adapted to the behavior you're expecting from.
05:21Uh we also allow model distillation.
05:23We do custom spec decoding for customers who need really advanced um custo custom spec decoding tooling.
05:30uh we do like on demand and like specific quantization and calibration of the model depending on your preferences and we do our our GPO and the fourth layer here is the deployment.
05:40So once you like run these models once you like train these models you need a way to deploy them.
05:45So these this is what allows us to push the models into production on our own um owned infra.
05:51So for most teams this is kind of usually a choice of different tools that you stitch together.
05:57But that's creates kind of a lot of friction and need for iteration.
06:00It slows the need for like the the ability to iterate here.
06:03And so what we allow is really to make the data flow from u the inference to the training and the training flows directly into production.
06:09So that allows you to keep like a virtuous loop of your product.
06:12And that's really what I think is the main difference between uh running a model and running an actual production uh AI system.
06:22Um so here is a little slide about what we handle so you don't have to.
06:26Um so running models in production is actually quite complicated.
06:29It takes a lot of work and so I think uh the way we approach it the value here is in three main layers.
06:34The first one is the optimization.
06:36So we actively tune how models run so the customers get much better speed and lower co and lower costs without having to figure out um all of that themselves.
06:45Like we really do the heavy lifting around the model deployment and optimization and really allow you to kind of uh come in and just play with them.
06:52This is like all around optimizing latency, throughput and cost at the actual engine level because you could be running the same open source model on different providers and have completely different um results in terms of costs.
07:03So there's a lot of technical stuff that goes on under the hood uh to make this work and Suji will cover that in a few minutes.
07:08Uh second layer is the infrastructure.
07:11So the fact that we actually own the hardware uh we keep it running.
07:15We own this like kind of vertical stack um gives us an edge that uh most providers don't have.
07:20So it's it's it's the control like we really control what the model does.
07:24Uh we handle like everything underneath and so we just give the the customers um the full control of the behavior and we take care of the infrastructure.
07:33Third layer is the enterprise readiness.
07:35I think this is really important here because like running open source seems kind of obvious but you need a bunch of ways to like secure like make it make sure it works in a secure environment.
07:44So there's a bunch of things here.
07:46Um security, compliance, reliability, they all requirements that all these large companies usually have.
07:51Um and they kind of need those before they can ship anything into production.
07:55And so a lot of AI providers make you figure out all of these yourself.
08:00And uh our mission here is to really handle all of these so you don't have to.
08:08Um something that comes up a lot with AI teams is that they start with inference.
08:12um they just get a model up and running and then they realize that pretty quickly quickly that running it is not just as easy as it seems and it's only the beginning of your path through running open source models at scale.
08:25So they really need to understand how it's actually performing in the real world and this is kind of where we come in.
08:31They want to improve it over time.
08:32Um they need to redeploy updates without breaking things.
08:35Um, and and that full cycle from running to observing to capturing and to redeploying the model is really what separates team from that ship like great AI products from team that ship demos.
08:45And I think that's really important to hear especially when talking about open source.
08:49Um, and so token factory covers that full loop uh inference to run the models.
08:53So as I said, Kimmy, Deepseek, all the all the main ones uh through real-time inference or batch inference through structured outputs and function calling um and on dedicated end points when it's necessary.
09:05Um and then moving on to the datab to collect and analyze the real world signal uh the completions, moving over to post training to improve and customize the model and finally moving to the last step and probably the most important one which is the deployment to push
09:18uh updates with full control over the model.
09:21And you actually really it's it's really important for most our customers to know where how it's running and what chips it's running on uh the level of quantization and all that stuff and that's really where we kind of operate uh and are comfortable.
09:33Most teams today we figured have to stitch different tools together to get this and we actually built uh one platform that tries and covers it end to end.
09:40So we really see this as a continuous improvement loop.
09:44Every cycle makes your AR more specific uh way faster and much cheaper.
09:50And now we'll dive into scaling open models to production.
09:54And I'll invite Suji on stage to take it over.
10:01All right.
10:02Thank you, Dylan.
10:04All right, everyone.
10:05So, um I'll zip through a few of these, but um I'm happy to um have a discussion afterwards.
10:10So, come and find us if you have any questions.
10:12So, we believe great inference starts with great models, but that's not enough.
10:19you need great infrastructure behind the scene to get the best performance and I believe that's what uh Nebio token fighter provides.
10:26Uh I get a lot of times I get asked like so this is great you guys host open models but how good are the open models really and I like to show you this benchmark here.
10:37Uh this is from artificial analysis.
10:40The black ones are proprietary models and the blue ones are open models.
10:44And the cool thing you can see here is how um the open models are actually very competitive sometimes even better than a lot of the proprietary models.
10:54So the gap between uh proprietary and open models is very narrow.
10:59Uh so we don't really have to chase proprietary models all the time for the intelligence.
11:04You can just just as well choose an open model that's just as smart and also ends up being much cheaper in most cases which is pretty cool.
11:12So what does it mean for customers?
11:14You get choice, right?
11:15You can run these open models in any provider, hopefully token factory, but so there's no vendor lock in and also you end up saving a lot of money on tokconomics.
11:26And here's a a few examples of models we host.
11:29Uh the top the top table is like the large models.
11:32We call them sort of the teacher models, right?
11:34They are very capable uh very smart but also kind of large.
11:38You can also use a large teacher models to train the smaller student models.
11:42So you sort of see the both large models and small models posted.
11:46So I want to sort of structure the talk a little bit um sort of the from the hardware layer, the serving layer and the optimizing layer.
11:54Uh so Dylan talked a little bit about hardware, we have we work very closely with Nvidia, we get the latest chipsets and uh not not just um the latest we also optimize
12:04uh kernels and runtimes.
12:06So the models run really uh really well.
12:08For example, something like the Nvidia floating point.
12:12standard we embraced it.
12:13It gives us a really good model performance on latest Nvidia chips.
12:17And if you think about model serving that kind of takes us to the middle layer and there are few optimization I'll talk about uh starting with the engines because you need an engine to serve the model.
12:29Uh we utilize a lot of the open source engines.
12:32Uh we have our own internal folk uh that we continuously tweak and optimize and we will deploy the model on the best engine possible.
12:40uh you know not all the models run equally well on all the engines.
12:43So we figure out which engines run well and we'll deploy the models accordingly.
12:47So as a customer you can just use an API not having to worry about you know which kind of GPU which engine because we take care of all that work behind the scene
12:58and uh when you have like hundreds of thousands of GPUs uh running models load balancing and routing becomes an issue.
13:05So a lot of the time people say oh that's easy right we just you know put a put a load balancer in front um send traffic randomly to different GPUs
13:13uh that's not how it works in LLMs because the workloads are slightly different for example just to kind of give you an idea uh in the early days of LLM
13:21the inputs were small I could just say write like a oneliner and the output is also small but now we are seeing your input could be pretty large like we are sending like large code bases
13:31uh as using coding agents to do like refactor or summarize or analyze right so the output as you can see can vary from like you know large to small large to large and routing them accordingly will give you um you know it's not it's not trivial
13:46so what we employ is we employ um routers that actually are cache aware what I mean by that is like if you look at the left side here you will see that my cache
13:56like different colors it's kind of fragmented all over so our cache rate is isn't all that great because my request landing randomly at different GPU use that may or may not have the cache.
14:07But if you look at the the one on the right, you can see that now the cache is much more coherent.
14:11See the colors are kind of together and the router knows where the caches are stored and it'll route them accordingly.
14:17So when we actually do the inference, you get pretty good cache and very good speed.
14:23Um another thing and this is something we are really excited about.
14:26Um it's called spec decoding.
14:29So the challenge is uh LLMs they generate tokens one by one one two three four right and then large models take a long time to generate these tokens.
14:40So the idea for spec decoding is how about we employ a smaller model which is a lot of the time faster and cheaper to generate the tokens but then have the large model verify at the end.
14:52So it's kind of like if you're like a senior engineer, you're kind of farming out the work to a junior engineer.
14:57So they're kind of doing the heavy lifting and then you are verifying the work.
15:00And this actually surprisingly scales really well.
15:03So the smaller models can generate tokens very quickly and the large model can verify and the and if it's good, we are done.
15:10If the large model doesn't like the the answer, it will actually regenerate the tokens for you.
15:15So that's kind of the worst case scenario.
15:17But most of the time you will get pretty good performance out of this by mixing small models and large models.
15:23And this actually a feature we actually baked into token factory.
15:26Uh that gives you pretty pretty good optimization um speed.
15:30So one of the things we experimented we sort of trained graph models.
15:34Uh you can train them using synthetic data.
15:36Even if you train them using like you know generic data you get like you know we seeing like up to 30% improvement.
15:42But it it gets even better if you can train the smaller model using your own data.
15:46Right?
15:47So we do a lot of experiments here and u we are launching a feature feature.
15:51You can actually train these draft models almost like with one click and you know we will capture your production data and train the draft models for you which is pretty cool.
16:00Uh another optimization uh behind the scene is KV cache.
16:03Um this is probably one of the biggest ROI in doing inference because uh creating tokens expensive especially when you're doing one at a time and the common sense is once you generate a token it it doesn't change.
16:17So there's no need to keep keep regenerating the same token over and over again.
16:21So the idea is we cache the tokens.
16:24So just to kind of give you an example here, let's say this is my prompt the cat sat on, right?
16:29Let's say the those are my prompts and I'm creating the next word and as soon as we create the next token, we cash it and the next time we need to look up, we don't have to regenerate regenerative, we just look up the cache very easy and this actually speeds up quite a bit.
16:44So just to kind of give you some ideas, we have seen uh speed ups anywhere from 5 to 10x, not just 5 to 10 percentage, actually 10x, right?
16:51because caching really works well.
16:53Um the thing about caching is it takes a lot of memory right because you know especially with our large inputs large token windows and one thing we we we've been implementing in token factory is
17:05when we do caching we can actually uh because GPU memory is very precious.
17:10So we can offload caching out of GPU memory automatically save it in a regular memory and then bring it back.
17:17So we don't like throw away the cache we spend a lot of time building.
17:20we actually offload it and bring it back and all of this is done automatically.
17:23So you don't have to worry about managing cache.
17:26This is done for you behind the scenes.
17:28So and this is one of the um um one of the important optimization we have done to speed up inference.
17:34Uh another one is called decoding.
17:36Um so the way you think about this is like there are two stages to LLM.
17:41So one of them is like sort of fill in the context that is very CPU intensive and then other one is actually decoding that is very memory intensive.
17:48And a lot of the time if you do them in the same GPU they're kind of computing for resources.
17:53So what we have done we sort of separate this two phases out.
17:56So one GPU set does prefill they are very you know uh compute efficient and the other one does uh boding that's very memory efficient and then they sort of transfer the KV cache in between and we have seen this is actually working working out pretty well.
18:12And another thing we do and and this is already you know already baked into our platform we also quantize the models uh and when we do quantizing you know we are reducing the precision.
18:23So if but we don't want to go too far right because then your model starts you know quality starts getting degrading.
18:28So what we do is we do a lot of experiments to figure out like what's the kind of the sweet spot of a quantization.
18:33So you can run the models efficiently but not not degrade the performance too much.
18:39So, so all of these things are already built in and and there's a there's a few more and interest of time I'll sort of wrap this up here and these are some of the things we kind of covered uh spec decoding KV cache and when you do caching doing um routing
18:54that is actually cache aware and also pre-fill uh you know uh separating prefill and decoding cases and if you're using an API you may not realize all these works going behind the scene
19:05but they are right and this is what make up you know make a a inference platform really great to serve especially the large large models we are seeing like the one trillion models we are seeing
19:16and finally I want to quickly um give this out so we um probably this is the most important slide of the presentation so we can take the take a quick picture or scan the code
19:27uh this will give you some credit on token factory platform so we allow you for allow for you guys to try it out and we have all the uh latest open source models uh GLM52
19:37kit K27s.
19:39Uh try them out.
19:40I've been using them them a lot a lot for coding lately and they work really really well.
19:45So if you're using a cursor or open code um you can very easily incorporate our models to um for coding work.
19:51Try them out.
19:52Let us know.
19:53And there's a discord channel um at the bottom.
19:55If you have any questions you can post them there or you can um you know um message me or Dylan.
20:00We will happy to um answer your questions.
20:02Uh so I'll stop here.
20:04We have like 10 seconds.
20:06[laughter]
20:07right on time, but uh we'll be around.
20:09Please come and talk to us if you have any questions.
20:11We'll happy to um chat more and then take any take your questions.
20:15Thank you all.