WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=TRe1u7dHYiA NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/TRe1u7dHYiA.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:01.309 --> 00:00:03.309 [music] s2 00:00:12.400 --> 00:00:13.920 Thank you for joining us today. s3 00:00:13.920 --> 00:00:14.960 Um I'm Dylan. s4 00:00:14.960 --> 00:00:17.359 I'm here today with my colleague Suji. s5 00:00:17.359 --> 00:00:28.240 So I lead the product marketing for Nibbius token factory and I'm here with Suji as a developer advocate and we're going to talk a bit about engineering um open NLMs for production. s6 00:00:28.240 --> 00:00:32.640 So before we jump right in, um I want to give a few words about what we're going to cover today. s7 00:00:32.640 --> 00:00:35.120 We only have 20 minutes, so we'll try to keep this brief. s8 00:00:35.120 --> 00:00:38.480 Uh but we're going to give you a quick intro on who we are, what we do. s9 00:00:38.480 --> 00:00:42.239 Uh then talk about what it takes to build the infra for inference. s10 00:00:42.239 --> 00:00:47.600 Um a few words on model shaping optimizations, which are extremely important in what we do. s11 00:00:47.600 --> 00:00:51.120 And then we'll talk about model serving optimizations. s12 00:00:52.719 --> 00:00:56.800 But before we jump right in, I just want to give a few words uh about who we are and what we do. s13 00:00:56.800 --> 00:00:59.039 So we work for Nibbius. s14 00:00:59.039 --> 00:01:05.199 Uh so specifically Nibbius token factory but Nibbius is a like a full stack AI cloud infrastructure. s15 00:01:05.199 --> 00:01:12.960 Um so this means we actually not only exposing model APIs we run the actual physical uh AI layer and infrastructure underneath it. s16 00:01:12.960 --> 00:01:23.119 Um we have our own data centers and video systems uh bare metal capacity manage cloud and optimizing the serving and the inference through Nibius token factory. s17 00:01:23.119 --> 00:01:26.880 Um top top row here is really about scale and trust. s18 00:01:26.880 --> 00:01:31.520 So we're a publicly traded company on the NASDAQ um headquartered in Amsterdam. s19 00:01:31.520 --> 00:01:42.720 Uh we're very close with Nvidia who made a 2 billion uh investment in Nebus a couple months ago and we're targeting a 5 gawit of Nvidia systems by the end of 2030. s20 00:01:42.720 --> 00:01:45.200 Uh bottom row is really more about the control of the stack. s21 00:01:45.200 --> 00:01:47.280 This is a quite a a good edge we have. s22 00:01:47.280 --> 00:01:53.439 We operate own data centers across the EU and the US uh bare metal rather than shared cloud. s23 00:01:53.439 --> 00:02:04.960 We're a very early adopter of Nvidia Rubin um CP like uh and bluefield storage and we were amongst the first European clouds to run both Blackwell Ultra AGXP300 and GB300 s24 00:02:04.960 --> 00:02:06.399 in production. s25 00:02:06.399 --> 00:02:15.920 Uh we also have um like strong contracts from Microsoft and Meta agreements that kind of show that the largest AR builders are already building this kind of capacity from us. s26 00:02:15.920 --> 00:02:18.160 Um and the last time is quite important. s27 00:02:18.160 --> 00:02:28.239 It's the uh like really the important bridge to this talk is the vertical integration from silicon to serve that we um rely on. s28 00:02:28.319 --> 00:02:30.480 So today we're talking about NEA token factory. s29 00:02:30.480 --> 00:02:34.239 So the idea here is to like really uh engineer AI for product. s30 00:02:34.239 --> 00:02:40.640 So we're managed inference company uh meaning we deploy these models and optimize them for our customers. s31 00:02:41.120 --> 00:02:44.959 So we noticed a few things today in the market of serving open source LLNs. s32 00:02:44.959 --> 00:02:49.680 Um, usually most AI teams are stuck between choosing two bad options. s33 00:02:49.680 --> 00:02:56.959 Um, closed APIs are really easy to start with, but you often hit a ceiling really fast and you can't really tune the model to your specific use case. s34 00:02:56.959 --> 00:02:59.280 Uh, you share the infra with everyone else. s35 00:02:59.280 --> 00:03:06.239 So, it kind of operates as a black box black box and the costs grow in a straight line with no way to really optimize things. s36 00:03:06.239 --> 00:03:08.319 The other option here is self-hosting. s37 00:03:08.319 --> 00:03:09.680 Obviously, it's quite fun. s38 00:03:09.680 --> 00:03:13.599 Uh, gives you full control on the model, but it's a massive engineering project. s39 00:03:13.599 --> 00:03:17.280 Um so you really need a dedicated team to actually keep it running. s40 00:03:17.280 --> 00:03:21.280 Um and you're usually month away from production before you even start your actual product. s41 00:03:21.280 --> 00:03:29.040 So token factory is kind of the shift uh this third path we identified uh before you even start like on your um actual product. s42 00:03:29.040 --> 00:03:33.599 You get the control and the performance of self-hosting with the simplicity of a manage inference service. s43 00:03:33.599 --> 00:03:39.440 So we do the heavy lifting, the hard infrastructure work and then customers stay focused on their product. s44 00:03:39.440 --> 00:03:44.000 This really unlocks uh three main things performance, cost and behavior of the models. s45 00:03:44.000 --> 00:03:54.720 Um as we've seen the gap between proprietary and open source models has been closing a lot and so this is where we kind of operate and we feel comfortable um operating is really not only s46 00:03:54.720 --> 00:04:03.920 relying on the fact that this bridge and this gap is closing down but also figuring out how you actually make it work in your own production uh use cases. s47 00:04:04.879 --> 00:04:06.720 So this is really the core of token factory. s48 00:04:06.720 --> 00:04:08.879 uh we've built a full stack JI platform. s49 00:04:08.879 --> 00:04:16.880 Um most platform we've seen on the market either give you inference training uh data tooling but here we decided to connect everything under one loop. s50 00:04:16.880 --> 00:04:23.840 So we have a bunch of tools to that kind of unlock this on the inference side um we allow you to run models in production. s51 00:04:23.840 --> 00:04:29.120 So we have 60 plus models on the platform JLM Kimmy DC you name them. s52 00:04:29.120 --> 00:04:34.160 Um we have dedicated endpoints we have different flavors on the models depending if you want to run them fast. s53 00:04:34.160 --> 00:04:38.720 uh we have different tooling like structured outputs, function calling and batch API. s54 00:04:38.720 --> 00:04:40.240 Second layer here is the data lab. s55 00:04:40.240 --> 00:04:43.199 So this allows you to really capture and structure the production logs. s56 00:04:43.199 --> 00:04:52.400 So we see a lot of customers actually running these models in production importing the completions and wanted a way to kind of slice and dice through these completions and this is really what we allow with the data lab. s57 00:04:52.400 --> 00:05:00.240 So uh inference log imports SQL data like uh data set filtering, data set versioning, um exporting and then batching. s58 00:05:00.240 --> 00:05:02.960 These uh third layer is post training. s59 00:05:02.960 --> 00:05:06.639 So we allow you to kind of fine-tune distill optimize these models. s60 00:05:06.639 --> 00:05:21.120 So this is a natural follow-up with the data lab is once you have your logs and your inference and you can generate synthetic data sets you can bring them all to post training and use that to kind of either through Laura or full fine tuning make the models much more adapted to the behavior you're expecting from. s61 00:05:21.120 --> 00:05:23.280 Uh we also allow model distillation. s62 00:05:23.280 --> 00:05:30.720 We do custom spec decoding for customers who need really advanced um custo custom spec decoding tooling. s63 00:05:30.720 --> 00:05:40.560 uh we do like on demand and like specific quantization and calibration of the model depending on your preferences and we do our our GPO and the fourth layer here is the deployment. s64 00:05:40.560 --> 00:05:45.360 So once you like run these models once you like train these models you need a way to deploy them. s65 00:05:45.360 --> 00:05:51.919 So these this is what allows us to push the models into production on our own um owned infra. s66 00:05:51.919 --> 00:05:57.120 So for most teams this is kind of usually a choice of different tools that you stitch together. s67 00:05:57.120 --> 00:06:00.639 But that's creates kind of a lot of friction and need for iteration. s68 00:06:00.639 --> 00:06:03.440 It slows the need for like the the ability to iterate here. s69 00:06:03.440 --> 00:06:09.520 And so what we allow is really to make the data flow from u the inference to the training and the training flows directly into production. s70 00:06:09.520 --> 00:06:12.720 So that allows you to keep like a virtuous loop of your product. s71 00:06:12.720 --> 00:06:20.160 And that's really what I think is the main difference between uh running a model and running an actual production uh AI system. s72 00:06:22.479 --> 00:06:26.160 Um so here is a little slide about what we handle so you don't have to. s73 00:06:26.160 --> 00:06:29.280 Um so running models in production is actually quite complicated. s74 00:06:29.280 --> 00:06:34.639 It takes a lot of work and so I think uh the way we approach it the value here is in three main layers. s75 00:06:34.639 --> 00:06:36.240 The first one is the optimization. s76 00:06:36.240 --> 00:06:45.440 So we actively tune how models run so the customers get much better speed and lower co and lower costs without having to figure out um all of that themselves. s77 00:06:45.440 --> 00:06:52.560 Like we really do the heavy lifting around the model deployment and optimization and really allow you to kind of uh come in and just play with them. s78 00:06:52.560 --> 00:07:03.360 This is like all around optimizing latency, throughput and cost at the actual engine level because you could be running the same open source model on different providers and have completely different um results in terms of costs. s79 00:07:03.360 --> 00:07:08.880 So there's a lot of technical stuff that goes on under the hood uh to make this work and Suji will cover that in a few minutes. s80 00:07:08.880 --> 00:07:11.120 Uh second layer is the infrastructure. s81 00:07:11.120 --> 00:07:15.039 So the fact that we actually own the hardware uh we keep it running. s82 00:07:15.039 --> 00:07:20.240 We own this like kind of vertical stack um gives us an edge that uh most providers don't have. s83 00:07:20.240 --> 00:07:24.960 So it's it's it's the control like we really control what the model does. s84 00:07:24.960 --> 00:07:33.759 Uh we handle like everything underneath and so we just give the the customers um the full control of the behavior and we take care of the infrastructure. s85 00:07:33.759 --> 00:07:35.680 Third layer is the enterprise readiness. s86 00:07:35.680 --> 00:07:44.800 I think this is really important here because like running open source seems kind of obvious but you need a bunch of ways to like secure like make it make sure it works in a secure environment. s87 00:07:44.800 --> 00:07:46.400 So there's a bunch of things here. s88 00:07:46.400 --> 00:07:51.599 Um security, compliance, reliability, they all requirements that all these large companies usually have. s89 00:07:51.599 --> 00:07:55.680 Um and they kind of need those before they can ship anything into production. s90 00:07:55.680 --> 00:08:00.080 And so a lot of AI providers make you figure out all of these yourself. s91 00:08:00.080 --> 00:08:06.240 And uh our mission here is to really handle all of these so you don't have to. s92 00:08:08.080 --> 00:08:12.400 Um something that comes up a lot with AI teams is that they start with inference. s93 00:08:12.400 --> 00:08:25.360 um they just get a model up and running and then they realize that pretty quickly quickly that running it is not just as easy as it seems and it's only the beginning of your path through running open source models at scale. s94 00:08:25.360 --> 00:08:31.039 So they really need to understand how it's actually performing in the real world and this is kind of where we come in. s95 00:08:31.039 --> 00:08:32.479 They want to improve it over time. s96 00:08:32.479 --> 00:08:35.680 Um they need to redeploy updates without breaking things. s97 00:08:35.680 --> 00:08:45.519 Um, and and that full cycle from running to observing to capturing and to redeploying the model is really what separates team from that ship like great AI products from team that ship demos. s98 00:08:45.519 --> 00:08:49.279 And I think that's really important to hear especially when talking about open source. s99 00:08:49.279 --> 00:08:53.760 Um, and so token factory covers that full loop uh inference to run the models. s100 00:08:53.760 --> 00:09:05.440 So as I said, Kimmy, Deepseek, all the all the main ones uh through real-time inference or batch inference through structured outputs and function calling um and on dedicated end points when it's necessary. s101 00:09:05.440 --> 00:09:18.399 Um and then moving on to the datab to collect and analyze the real world signal uh the completions, moving over to post training to improve and customize the model and finally moving to the last step and probably the most important one which is the deployment to push s102 00:09:18.399 --> 00:09:21.120 uh updates with full control over the model. s103 00:09:21.120 --> 00:09:33.200 And you actually really it's it's really important for most our customers to know where how it's running and what chips it's running on uh the level of quantization and all that stuff and that's really where we kind of operate uh and are comfortable. s104 00:09:33.200 --> 00:09:40.959 Most teams today we figured have to stitch different tools together to get this and we actually built uh one platform that tries and covers it end to end. s105 00:09:40.959 --> 00:09:44.320 So we really see this as a continuous improvement loop. s106 00:09:44.320 --> 00:09:50.320 Every cycle makes your AR more specific uh way faster and much cheaper. s107 00:09:50.800 --> 00:09:54.000 And now we'll dive into scaling open models to production. s108 00:09:54.000 --> 00:09:57.920 And I'll invite Suji on stage to take it over. s109 00:10:01.920 --> 00:10:02.320 All right. s110 00:10:02.320 --> 00:10:04.640 Thank you, Dylan. s111 00:10:04.800 --> 00:10:05.519 All right, everyone. s112 00:10:05.519 --> 00:10:10.080 So, um I'll zip through a few of these, but um I'm happy to um have a discussion afterwards. s113 00:10:10.080 --> 00:10:12.880 So, come and find us if you have any questions. s114 00:10:12.880 --> 00:10:19.120 So, we believe great inference starts with great models, but that's not enough. s115 00:10:19.120 --> 00:10:26.959 you need great infrastructure behind the scene to get the best performance and I believe that's what uh Nebio token fighter provides. s116 00:10:26.959 --> 00:10:37.760 Uh I get a lot of times I get asked like so this is great you guys host open models but how good are the open models really and I like to show you this benchmark here. s117 00:10:37.760 --> 00:10:40.160 Uh this is from artificial analysis. s118 00:10:40.160 --> 00:10:44.800 The black ones are proprietary models and the blue ones are open models. s119 00:10:44.800 --> 00:10:54.399 And the cool thing you can see here is how um the open models are actually very competitive sometimes even better than a lot of the proprietary models. s120 00:10:54.399 --> 00:10:59.440 So the gap between uh proprietary and open models is very narrow. s121 00:10:59.440 --> 00:11:04.240 Uh so we don't really have to chase proprietary models all the time for the intelligence. s122 00:11:04.240 --> 00:11:12.800 You can just just as well choose an open model that's just as smart and also ends up being much cheaper in most cases which is pretty cool. s123 00:11:12.800 --> 00:11:14.560 So what does it mean for customers? s124 00:11:14.560 --> 00:11:15.600 You get choice, right? s125 00:11:15.600 --> 00:11:25.920 You can run these open models in any provider, hopefully token factory, but so there's no vendor lock in and also you end up saving a lot of money on tokconomics. s126 00:11:26.399 --> 00:11:29.360 And here's a a few examples of models we host. s127 00:11:29.360 --> 00:11:32.320 Uh the top the top table is like the large models. s128 00:11:32.320 --> 00:11:34.240 We call them sort of the teacher models, right? s129 00:11:34.240 --> 00:11:38.079 They are very capable uh very smart but also kind of large. s130 00:11:38.079 --> 00:11:42.240 You can also use a large teacher models to train the smaller student models. s131 00:11:42.240 --> 00:11:46.160 So you sort of see the both large models and small models posted. s132 00:11:46.160 --> 00:11:54.320 So I want to sort of structure the talk a little bit um sort of the from the hardware layer, the serving layer and the optimizing layer. s133 00:11:54.320 --> 00:12:04.640 Uh so Dylan talked a little bit about hardware, we have we work very closely with Nvidia, we get the latest chipsets and uh not not just um the latest we also optimize s134 00:12:04.640 --> 00:12:06.560 uh kernels and runtimes. s135 00:12:06.560 --> 00:12:08.959 So the models run really uh really well. s136 00:12:08.959 --> 00:12:12.000 For example, something like the Nvidia floating point. s137 00:12:12.000 --> 00:12:13.279 standard we embraced it. s138 00:12:13.279 --> 00:12:17.519 It gives us a really good model performance on latest Nvidia chips. s139 00:12:17.519 --> 00:12:29.440 And if you think about model serving that kind of takes us to the middle layer and there are few optimization I'll talk about uh starting with the engines because you need an engine to serve the model. s140 00:12:29.440 --> 00:12:32.240 Uh we utilize a lot of the open source engines. s141 00:12:32.240 --> 00:12:40.720 Uh we have our own internal folk uh that we continuously tweak and optimize and we will deploy the model on the best engine possible. s142 00:12:40.720 --> 00:12:43.680 uh you know not all the models run equally well on all the engines. s143 00:12:43.680 --> 00:12:47.680 So we figure out which engines run well and we'll deploy the models accordingly. s144 00:12:47.680 --> 00:12:57.600 So as a customer you can just use an API not having to worry about you know which kind of GPU which engine because we take care of all that work behind the scene s145 00:12:58.000 --> 00:13:05.360 and uh when you have like hundreds of thousands of GPUs uh running models load balancing and routing becomes an issue. s146 00:13:05.360 --> 00:13:13.120 So a lot of the time people say oh that's easy right we just you know put a put a load balancer in front um send traffic randomly to different GPUs s147 00:13:13.120 --> 00:13:21.519 uh that's not how it works in LLMs because the workloads are slightly different for example just to kind of give you an idea uh in the early days of LLM s148 00:13:21.519 --> 00:13:31.920 the inputs were small I could just say write like a oneliner and the output is also small but now we are seeing your input could be pretty large like we are sending like large code bases s149 00:13:31.920 --> 00:13:46.240 uh as using coding agents to do like refactor or summarize or analyze right so the output as you can see can vary from like you know large to small large to large and routing them accordingly will give you um you know it's not it's not trivial s150 00:13:46.240 --> 00:13:56.160 so what we employ is we employ um routers that actually are cache aware what I mean by that is like if you look at the left side here you will see that my cache s151 00:13:56.160 --> 00:14:07.040 like different colors it's kind of fragmented all over so our cache rate is isn't all that great because my request landing randomly at different GPU use that may or may not have the cache. s152 00:14:07.040 --> 00:14:11.360 But if you look at the the one on the right, you can see that now the cache is much more coherent. s153 00:14:11.360 --> 00:14:17.279 See the colors are kind of together and the router knows where the caches are stored and it'll route them accordingly. s154 00:14:17.279 --> 00:14:23.199 So when we actually do the inference, you get pretty good cache and very good speed. s155 00:14:23.839 --> 00:14:26.720 Um another thing and this is something we are really excited about. s156 00:14:26.720 --> 00:14:29.199 Um it's called spec decoding. s157 00:14:29.199 --> 00:14:40.160 So the challenge is uh LLMs they generate tokens one by one one two three four right and then large models take a long time to generate these tokens. s158 00:14:40.160 --> 00:14:52.959 So the idea for spec decoding is how about we employ a smaller model which is a lot of the time faster and cheaper to generate the tokens but then have the large model verify at the end. s159 00:14:52.959 --> 00:14:57.519 So it's kind of like if you're like a senior engineer, you're kind of farming out the work to a junior engineer. s160 00:14:57.519 --> 00:15:00.959 So they're kind of doing the heavy lifting and then you are verifying the work. s161 00:15:00.959 --> 00:15:03.519 And this actually surprisingly scales really well. s162 00:15:03.519 --> 00:15:10.639 So the smaller models can generate tokens very quickly and the large model can verify and the and if it's good, we are done. s163 00:15:10.639 --> 00:15:15.680 If the large model doesn't like the the answer, it will actually regenerate the tokens for you. s164 00:15:15.680 --> 00:15:17.519 So that's kind of the worst case scenario. s165 00:15:17.519 --> 00:15:23.440 But most of the time you will get pretty good performance out of this by mixing small models and large models. s166 00:15:23.440 --> 00:15:26.720 And this actually a feature we actually baked into token factory. s167 00:15:26.720 --> 00:15:30.800 Uh that gives you pretty pretty good optimization um speed. s168 00:15:30.800 --> 00:15:34.480 So one of the things we experimented we sort of trained graph models. s169 00:15:34.480 --> 00:15:36.959 Uh you can train them using synthetic data. s170 00:15:36.959 --> 00:15:42.160 Even if you train them using like you know generic data you get like you know we seeing like up to 30% improvement. s171 00:15:42.160 --> 00:15:46.959 But it it gets even better if you can train the smaller model using your own data. s172 00:15:46.959 --> 00:15:47.279 Right? s173 00:15:47.279 --> 00:15:51.519 So we do a lot of experiments here and u we are launching a feature feature. s174 00:15:51.519 --> 00:16:00.399 You can actually train these draft models almost like with one click and you know we will capture your production data and train the draft models for you which is pretty cool. s175 00:16:00.399 --> 00:16:03.680 Uh another optimization uh behind the scene is KV cache. s176 00:16:03.680 --> 00:16:17.839 Um this is probably one of the biggest ROI in doing inference because uh creating tokens expensive especially when you're doing one at a time and the common sense is once you generate a token it it doesn't change. s177 00:16:17.839 --> 00:16:21.839 So there's no need to keep keep regenerating the same token over and over again. s178 00:16:21.839 --> 00:16:24.079 So the idea is we cache the tokens. s179 00:16:24.079 --> 00:16:29.199 So just to kind of give you an example here, let's say this is my prompt the cat sat on, right? s180 00:16:29.199 --> 00:16:44.000 Let's say the those are my prompts and I'm creating the next word and as soon as we create the next token, we cash it and the next time we need to look up, we don't have to regenerate regenerative, we just look up the cache very easy and this actually speeds up quite a bit. s181 00:16:44.000 --> 00:16:51.600 So just to kind of give you some ideas, we have seen uh speed ups anywhere from 5 to 10x, not just 5 to 10 percentage, actually 10x, right? s182 00:16:51.600 --> 00:16:53.839 because caching really works well. s183 00:16:53.839 --> 00:17:05.039 Um the thing about caching is it takes a lot of memory right because you know especially with our large inputs large token windows and one thing we we we've been implementing in token factory is s184 00:17:05.039 --> 00:17:10.079 when we do caching we can actually uh because GPU memory is very precious. s185 00:17:10.079 --> 00:17:17.280 So we can offload caching out of GPU memory automatically save it in a regular memory and then bring it back. s186 00:17:17.280 --> 00:17:20.480 So we don't like throw away the cache we spend a lot of time building. s187 00:17:20.480 --> 00:17:23.600 we actually offload it and bring it back and all of this is done automatically. s188 00:17:23.600 --> 00:17:26.079 So you don't have to worry about managing cache. s189 00:17:26.079 --> 00:17:28.480 This is done for you behind the scenes. s190 00:17:28.480 --> 00:17:34.960 So and this is one of the um um one of the important optimization we have done to speed up inference. s191 00:17:34.960 --> 00:17:36.880 Uh another one is called decoding. s192 00:17:36.880 --> 00:17:41.039 Um so the way you think about this is like there are two stages to LLM. s193 00:17:41.039 --> 00:17:48.559 So one of them is like sort of fill in the context that is very CPU intensive and then other one is actually decoding that is very memory intensive. s194 00:17:48.559 --> 00:17:53.200 And a lot of the time if you do them in the same GPU they're kind of computing for resources. s195 00:17:53.200 --> 00:17:56.480 So what we have done we sort of separate this two phases out. s196 00:17:56.480 --> 00:18:11.280 So one GPU set does prefill they are very you know uh compute efficient and the other one does uh boding that's very memory efficient and then they sort of transfer the KV cache in between and we have seen this is actually working working out pretty well. s197 00:18:12.240 --> 00:18:23.440 And another thing we do and and this is already you know already baked into our platform we also quantize the models uh and when we do quantizing you know we are reducing the precision. s198 00:18:23.440 --> 00:18:28.799 So if but we don't want to go too far right because then your model starts you know quality starts getting degrading. s199 00:18:28.799 --> 00:18:33.760 So what we do is we do a lot of experiments to figure out like what's the kind of the sweet spot of a quantization. s200 00:18:33.760 --> 00:18:39.120 So you can run the models efficiently but not not degrade the performance too much. s201 00:18:39.120 --> 00:18:54.720 So, so all of these things are already built in and and there's a there's a few more and interest of time I'll sort of wrap this up here and these are some of the things we kind of covered uh spec decoding KV cache and when you do caching doing um routing s202 00:18:54.720 --> 00:19:05.520 that is actually cache aware and also pre-fill uh you know uh separating prefill and decoding cases and if you're using an API you may not realize all these works going behind the scene s203 00:19:05.520 --> 00:19:16.160 but they are right and this is what make up you know make a a inference platform really great to serve especially the large large models we are seeing like the one trillion models we are seeing s204 00:19:16.160 --> 00:19:27.679 and finally I want to quickly um give this out so we um probably this is the most important slide of the presentation so we can take the take a quick picture or scan the code s205 00:19:27.679 --> 00:19:37.200 uh this will give you some credit on token factory platform so we allow you for allow for you guys to try it out and we have all the uh latest open source models uh GLM52 s206 00:19:37.200 --> 00:19:39.200 kit K27s. s207 00:19:39.200 --> 00:19:40.160 Uh try them out. s208 00:19:40.160 --> 00:19:45.039 I've been using them them a lot a lot for coding lately and they work really really well. s209 00:19:45.039 --> 00:19:51.919 So if you're using a cursor or open code um you can very easily incorporate our models to um for coding work. s210 00:19:51.919 --> 00:19:52.559 Try them out. s211 00:19:52.559 --> 00:19:53.360 Let us know. s212 00:19:53.360 --> 00:19:55.280 And there's a discord channel um at the bottom. s213 00:19:55.280 --> 00:20:00.160 If you have any questions you can post them there or you can um you know um message me or Dylan. s214 00:20:00.160 --> 00:20:02.640 We will happy to um answer your questions. s215 00:20:02.640 --> 00:20:04.240 Uh so I'll stop here. s216 00:20:04.240 --> 00:20:06.523 We have like 10 seconds. s217 00:20:06.523 --> 00:20:07.520 [laughter] s218 00:20:07.520 --> 00:20:09.600 right on time, but uh we'll be around. s219 00:20:09.600 --> 00:20:11.360 Please come and talk to us if you have any questions. s220 00:20:11.360 --> 00:20:15.039 We'll happy to um chat more and then take any take your questions. s221 00:20:15.039 --> 00:20:17.440 Thank you all.