When Running Your Own AI Models Beats Paying for APIs - Steven Latré, imec
https://www.youtube.com/watch?v=9-Y_oS9OBLc
AI Engineer · 18 min · 165 sentences · sentence-level transcript from YouTube's caption track; each line begins at the time shown

[00:12] So my name is Steven.
[00:13] I'm leading the AI department at IMEC.
[00:17] For those of you who don't know what IMEC is, we're not that well known actually in the AI space, but we're very well known known in the semiconductor space.
[00:24] Been seen a little bit as the the hidden gem.
[00:27] We exist already for about more than four decades and what we typically do is we have the most advanced R&D clean room in the world and we actually run
[00:36] together with our partners next generation semiconductors in that sense.
[00:40] And so actually the impact that we've had in that semiconductor space is pretty huge.
[00:44] I think every single transistor technology that was more or less that is deployed now has been invented by us in the last four four decades in that that sense.
[00:54] So in that sense pretty much a hidden gem in in in Belgium where I'm where I'm from.
[00:58] So why is a a semiconductor nerd talking about artificial intelligence?
[01:04] Well, we do way more than just transistor technology.
[01:06] We go all the way up to systems.
[01:08] We go all the way up to the workloads as as well and that's where I'm responsible from for where we're trying to connect what AI workloads does to infrastructure
[01:18] in that that sense.
[01:20] On the one hand because we've lived that as well the last few few years.
[01:24] Of course within IMEC we're also working very much with with AI to kind of realize agentica flows as as well.
[01:32] And what we've experienced I think a lot of you of us have experienced at the last few few years is kind of a different stages of AI maturity.
[01:39] We we kind of started off in 2024 2025 thinking that we needed the latest and the greatest of every single frontier model and that the best wasn't good enough and that we needed more and more.
[01:52] But of course at the same time what we realized is that prices started evolve.
[01:57] We moved from more subscription based pricing to more token-based uh pricing.
[02:03] And then like the last year, we moved more and more towards model routing where we try to find the best type of model for the task at at hand.
[02:14] Things that I think all of us have been have been doing the last year or so as as as well.
[02:20] What we're actually seeing as well with both within the company but also with the customers that we're that we're talking about is that there's other stages to that as as well.
[02:29] Model routing is in that sense maybe the first just low-hanging fruits in in getting a good optimization of finding the right model to the right task.
[02:38] We're seeing more and more as as token consumption really really grows that there's a need for doing other things as well and doing other things basically means taking control of your own stack in that that cell.
[02:51] That sense.
[02:52] So we're seeing customers and we're as on that path as well moving from well, first of all, we started with just looking at closed APIs, just grabbing what we could do.
[03:01] Then second phase was for us more, okay, let's see how we can do cost optimization.
[03:07] We're very much ourself now in the the third phase where we're looking at how can we host open weights models and how can we lease our infrastructure in the long run maybe even
[03:18] buy our infrastructure as as well.
[03:22] I particularly want to talk about this third and fourth phase of of of all of this and see what it actually would would mean because running an API or just having an API
[03:32] for to for token consumption is pretty damn straightforward.
[03:36] It's pretty easy to to do but the moment you move towards this third and fourth phase, suddenly the design space starts blowing up as as well.
[03:45] In the first phase, we're just worried about what is the right type of model for the right type of model accuracy or for the right type of task complexity.
[03:53] But the moment you start owning your your infrastructure, you suddenly need to worry about a whole lot of other things as as well.
[04:00] What kind of model am I going to take?
[04:02] How can I trust this this model?
[04:04] Uh, you need to think about harnesses, but you also need to think about a lot of hardware configurations as as as well, such as what is the right hardware architecture that I would would use for for this?
[04:15] What is the CPU to GPU balance?
[04:17] How can I do KB cache optimization and uh, and all of that.
[04:22] And so that's where we want to position ourselves as I am going forward as well, where of course here in a in a world that is very much aware of models and harnesses, but we do see very much also a gap from those models and harnesses
[04:33] to that infrastructure as as as well.
[04:37] Um, so what we did in the last few months and what I'm here to present today is a kind of benchmarking study that we did in the last few months.
[04:44] We did uh, really thousands of of runs to really really quantify what does that move from going from an API to really infrastructure really uh, give you.
[04:55] And so really from an Agentic AI perspective.
[04:57] So if you have an Agentic AI task, if you have a specific Agentic AI task performance, what would be the cost for you if you move from an API
[05:05] towards your own uh, infrastructure?
[05:08] So that also means moving from a more token-based cost subscription to really kind of total cost of of ownership.
[05:14] And also um, how should you then configure the infrastructure?
[05:18] Really from what kind of harnesses are relevant?
[05:20] Uh, what kind of hardware would you need?
[05:22] What is the inference engine of choice uh, for for all of this?
[05:27] So how did we do this?
[05:28] Uh, we really wanted to compare at a task level, so we looked at uh, long horizon tasks specifically because long horizon tasks are really the perfect uh, um, if you really have your own infrastructure that you want to uh, to own.
[05:41] If you do do it very very interactive, it might not be the economies might not be the same, but for long horizon tasks, we think that that's the first reason or the first pivot point where um, taking
[05:52] a hold of your AI stack is really relevant.
[05:55] So we took one specific one being um, uh, the SWE-bench pro uh, tasks, which is set of benchmark tasks that exist um, uh, for coding assistance.
[06:06] And what we did is we kind of validate or kind of very different types of coding harnesses.
[06:11] We took Codex by and and Cloud Code for this, which are more or less the three most popular harnesses out there for for coding assistance.
[06:19] And then we kind of vary the different types of models that we could use on this as a as well.
[06:24] Starting with a traditional standard API environment, [clears throat] but then also looking at a wide range of different open weight models with different types of complexities and and parameter sizes as a as well.
[06:39] But as this is a talk about translate translating this towards your own infrastructure, we of course also map this to the right hardware architecture at hand as well.
[06:48] Always trying to find the right type architecture that kind of is still sufficient enough to really do this at at at scale and it where you really have where you can have multiple concurrency
[06:58] concurrent users on on this.
[07:01] So in essence, we kind of increased the hardware complexity, increased the model complexity, but at the same time also looked at really the application performance of all of this.
[07:09] So what is the kind of the resolve rate of your task?
[07:12] What is the latency and what is the cost?
[07:14] All in all in all this was more than a thousands of different runs that we did.
[07:19] We set up a full benchmarking pipeline and I kind of want to run you run you down with the different results we got out of out of this.
[07:28] So let's start maybe with the most obvious one.
[07:30] Kind of different models, different model complexities, different model parameters sizes actually give you different accuracies of these of these tasks.
[07:41] We could actually really classify three main categories of of models.
[07:47] The more smaller ones and with smaller ones I would say up to a few hundred billion of parameters where Quen and and DeepSeeker are part of.
[07:57] They kind of got a got into a resolve rate of about 32 5 to 40% of tasks that really were able to achieve accurately correctly.
[08:08] Second one like GLM, um, was really at Opus level, uh, uh, range and then Kimik 3 that came out a few months ago, um, was, uh, really was really crushing it to,
[08:20] in all honesty, also probably because part of it has been probably in the training set as well, but still the the the results were quite, uh, substantial.
[08:28] Had a resolve rate of 86%.
[08:30] Now, all of this is not something that uh, is is something that you can easily find on artificial analysis as well.
[08:36] This doesn't really give you a lot of uh, uh, additional insights.
[08:40] What I think is really more interesting is if you add the infrastructure and hardware angle to it as well.
[08:45] And there we do see quite a bit of of of differences.
[08:48] What I think is interesting is if you compare for example Qwen with with DeepSeek that all in all the application performance of it is more or less on the same,
[08:58] uh, level.
[08:59] Although if you then look at what kind of hardware you would need for Qwen and DeepSeek, you're looking at a factor of four and in terms of renting GPUs that are is more expensive.
[09:08] Um, similarly if you just make the move from, um, a GLM model to, uh, a Kimik 3, uh, model, it gives you a bump of about 20% in in accuracy, but all in all it's just giving you only a 15%
[09:21] increase in cost in renting GPUs as well.
[09:24] So what does that tell you?
[09:25] It basically tells you that if you move this make this move towards open weight models that it's super super important to kind of look at the hardware constraints of this as well and also look look at the actual,
[09:38] um, costs of, uh, of all of this and there's definitely not a linear curve to be found here.
[09:45] So often it's just a step wise function where you where it might be that you're choosing a specific, uh, model and that just taking a slightly better or slightly,
[09:55] uh, less performant model would have a huge impact on the hardware architecture of choice as as well.
[10:02] So that's one and that's looking at mainly from an application performance perspective.
[10:06] Um, let's take a different angle to it and that's concurrency.
[10:10] Um because when you're using an API, um you can um get a certain performance of of token consumption.
[10:19] You know, you know more or less what your token latency is going to be, and you know what your token's throughput is going to be.
[10:24] The moment you make the move towards a more infrastructure-based solution, suddenly you need to worry about concurrency as well.
[10:31] In other words, how many users can I reliably uh put on the same uh machine without the system completely deteriorating from a latency perspective in that that sense.
[10:42] There's a a memory size to it, but there's also a latency uh perspective to it.
[10:47] Uh memory means that uh on the one hand, the most systems are memory bound, and it means that on the one hand, your model parameter size need to fit your your your memory, but also your user's KV cache as a as well.
[11:00] And as you have an increasing number of um of users uh using this the system, your KV cache uh requirement will become bigger and bigger as a as well.
[11:10] So, what we did is we kind of tried to look at the different types of solutions I just already showed you, and look at how many concurrent users we were actually able to fit really comfortably
[11:19] in that uh that system.
[11:21] And the results were pretty um uh surprising, or at least also maybe not that positive for really the bigger models as a as as well.
[11:31] So, bear in mind, each of these different models have their own hardware architecture that is uh that is optimal for it.
[11:36] But if you then look at the comfortable concurrency levels that we were able to get, it really really dropped down very significantly, even with bigger hardware architectures as um as well.
[11:47] And in all cases, we were much slower than uh uh our baseline, which was uh Cloud Code in that that sense.
[11:55] By the way, I think you figured it all out already.
[11:57] Um we actually have a detailed blog post on each of the topics I'm I'm talking about.
[12:01] Every QR code actually leads you to to that as um as as well.
[12:06] So, this kind of shows you that uh concurrency is actually probably even more important than accuracy in the in this sense.
[12:13] Accuracy gives you the kind of baseline, but then even for the really big models and even if especially if you look at very long horizon tasks like code assistance,
[12:23] running a Kimi K3 model only gives you about four concurrent users that can really run reliably on a on this and still the latency is still a factor three slower.
[12:37] So, what we then did is a kind of full sweet spots analysis of okay, in which cases is it really interesting to stick to your API usage?
[12:46] In which cases would it make sense more for you to kind of move to renting out GPUs with the the various model providers that are out there and in which case would you really
[12:58] go for would it be more interesting for purchasing and really investing in your own infrastructure and building it out on prem.
[13:06] Of course, there can be a lot of other reasons as well to make that that switch but if we look at it from a total cost of ownership perspective.
[13:14] So, without going into too much details what we saw there is that the lower your hardware architecture system actually is, the worse it is to move from API to your own infrastructure.
[13:25] We saw that if you really are at the lower tiers of of of models and as a result also at the lower tiers of the hardware architecture that you would need,
[13:33] you actually need an GPU utilization of about 89% so that means that your GPU needs to run at capacity at 89% of the of the time.
[13:45] Um, similarly if you actually move to more bigger hardware systems and I made a mistake there it should be a higher hardware systems.
[13:52] So, for example an 8 times B200s, you're okay with just 15% of of GPU utilization and that's really realistic and 89% is just completely infeasible in that that sense.
[14:05] If you then kind of average it out and if you look at for example GLM class applications, what we do see is that that's really the sweet spot or the threshold of when
[14:15] moving from API to infrastructure makes it more interesting as a as well.
[14:20] And in the full total cost of ownership analysis that we did specifically for this application being coding assistance, we really saw that you get into a kind of 24 5%
[14:30] of of cost saving.
[14:33] So that's really infrastructure perspective.
[14:34] Let's kind of switch it over to harnesses.
[14:37] We also varied the amount the harnesses as a as as well and looked at what the application performance was for those three different harnesses.
[14:45] And on the first glance, the application performance actually didn't really change that much, which was maybe kind of surprising.
[14:52] Every single task we we gave every single harness we gave you more or less got it and every single for different kind of models, we kind of got into the same kind of application accuracy on this
[15:03] on this benchmark.
[15:03] So at the first glance, you could say, well, the harness actually doesn't have an impact whatsoever on that because it gets the job done.
[15:11] Well, that might be the case if you're going for an API only usage, but it's completely different if you're moving towards your own infrastructure and you actually pay for the GPU
[15:21] usage because then the wall time of your GPU really is is relevant.
[15:25] And what we saw there is that different kinds of harnesses actually had a huge impact on the amount of token consumption we had in the in this and then specifically
[15:35] in the case of of Codex, we had for a GLM only half of the of tokens that were actually being being generated.
[15:44] And as a result, also the wall time of using these GPUs actually was half of that as well.
[15:50] So that means that although the harnesses actually gave you the right type of application performance or more or less a similar level of application performance, the GPU bill you would in the end have would be half
[16:03] of of the other harnesses as well.
[16:06] So it really shows very clearly that in that case also the right harness to model choice might not have an impact on application performance, but it has really an impact
[16:15] on your total cost of ownership for your GPUs as a as well.
[16:20] So, this is just the first study that we're that we did in the last few few months on this.
[16:27] Where are we going with this?
[16:27] I think there's a lot to unpack.
[16:31] It's very clear that moving this from API to infrastructure that is just not very simple or very linear choice.
[16:38] And I have showed this now for one single application being coding assistance, but a real enterprise mix would often have a combination of a lot of difference of applications
[16:49] as as as well.
[16:50] And that actually complicates things even even further.
[16:54] We also saw that there's a very complex relationship between harnesses, between the models, between the infrastructure.
[16:59] And I haven't touched upon other things like what do we do in terms of quantization?
[17:04] What is the impact of specific optimizations like speculative decoding that we applied as well, but that we haven't configured into to detail.
[17:13] And the third dimension is of course that it's very much a constantly moving target as well.
[17:17] New models are being being launched.
[17:20] New hardware architecture choices are there as as well.
[17:24] So, we're actually building out a kind of benchmarking pipeline that we already have.
[17:30] We're building this out to a full benchmarking data set as well that we're going to share publicly for for everyone and trying to keep track of this moving target
[17:40] moving moving forward.
[17:42] But at the same time we cannot do this alone.
[17:45] We're a research organization.
[17:46] We're a non-profit research organization and the biggest important part for us is trying to bring that ecosystem together.
[17:53] And I think that the software world and the hardware world deserves to be much closer together as well.
[18:00] So, we're actually launching a kind of alliance to bring people who are interested in this part, software people from the software who are interested in hardware optimization.
[18:09] We're bringing this together with our existing customer portfolio of new and existing chip providers and infrastructure providers as well.
[18:17] And if you would be interested in this, uh free to reach out, uh after this uh this talk.
[18:23] Thanks a lot.