When Running Your Own AI Models Beats Paying for APIs - Steven Latré, imec

AI Engineer · 18 min · 165 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:12So my name is Steven.
00:13I'm leading the AI department at IMEC.
00:17For those of you who don't know what IMEC is, we're not that well known actually in the AI space, but we're very well known known in the semiconductor space.
00:24Been seen a little bit as the the hidden gem.
00:27We exist already for about more than four decades and what we typically do is we have the most advanced R&D clean room in the world and we actually run
00:36together with our partners next generation semiconductors in that sense.
00:40And so actually the impact that we've had in that semiconductor space is pretty huge.
00:44I think every single transistor technology that was more or less that is deployed now has been invented by us in the last four four decades in that that sense.
00:54So in that sense pretty much a hidden gem in in in Belgium where I'm where I'm from.
00:58So why is a a semiconductor nerd talking about artificial intelligence?
01:04Well, we do way more than just transistor technology.
01:06We go all the way up to systems.
01:08We go all the way up to the workloads as as well and that's where I'm responsible from for where we're trying to connect what AI workloads does to infrastructure
01:18in that that sense.
01:20On the one hand because we've lived that as well the last few few years.
01:24Of course within IMEC we're also working very much with with AI to kind of realize agentica flows as as well.
01:32And what we've experienced I think a lot of you of us have experienced at the last few few years is kind of a different stages of AI maturity.
01:39We we kind of started off in 2024 2025 thinking that we needed the latest and the greatest of every single frontier model and that the best wasn't good enough and that we needed more and more.
01:52But of course at the same time what we realized is that prices started evolve.
01:57We moved from more subscription based pricing to more token-based uh pricing.
02:03And then like the last year, we moved more and more towards model routing where we try to find the best type of model for the task at at hand.
02:14Things that I think all of us have been have been doing the last year or so as as as well.
02:20What we're actually seeing as well with both within the company but also with the customers that we're that we're talking about is that there's other stages to that as as well.
02:29Model routing is in that sense maybe the first just low-hanging fruits in in getting a good optimization of finding the right model to the right task.
02:38We're seeing more and more as as token consumption really really grows that there's a need for doing other things as well and doing other things basically means taking control of your own stack in that that cell.
02:51That sense.
02:52So we're seeing customers and we're as on that path as well moving from well, first of all, we started with just looking at closed APIs, just grabbing what we could do.
03:01Then second phase was for us more, okay, let's see how we can do cost optimization.
03:07We're very much ourself now in the the third phase where we're looking at how can we host open weights models and how can we lease our infrastructure in the long run maybe even
03:18buy our infrastructure as as well.
03:22I particularly want to talk about this third and fourth phase of of of all of this and see what it actually would would mean because running an API or just having an API
03:32for to for token consumption is pretty damn straightforward.
03:36It's pretty easy to to do but the moment you move towards this third and fourth phase, suddenly the design space starts blowing up as as well.
03:45In the first phase, we're just worried about what is the right type of model for the right type of model accuracy or for the right type of task complexity.
03:53But the moment you start owning your your infrastructure, you suddenly need to worry about a whole lot of other things as as well.
04:00What kind of model am I going to take?
04:02How can I trust this this model?
04:04Uh, you need to think about harnesses, but you also need to think about a lot of hardware configurations as as as well, such as what is the right hardware architecture that I would would use for for this?
04:15What is the CPU to GPU balance?
04:17How can I do KB cache optimization and uh, and all of that.
04:22And so that's where we want to position ourselves as I am going forward as well, where of course here in a in a world that is very much aware of models and harnesses, but we do see very much also a gap from those models and harnesses
04:33to that infrastructure as as as well.
04:37Um, so what we did in the last few months and what I'm here to present today is a kind of benchmarking study that we did in the last few months.
04:44We did uh, really thousands of of runs to really really quantify what does that move from going from an API to really infrastructure really uh, give you.
04:55And so really from an Agentic AI perspective.
04:57So if you have an Agentic AI task, if you have a specific Agentic AI task performance, what would be the cost for you if you move from an API
05:05towards your own uh, infrastructure?
05:08So that also means moving from a more token-based cost subscription to really kind of total cost of of ownership.
05:14And also um, how should you then configure the infrastructure?
05:18Really from what kind of harnesses are relevant?
05:20Uh, what kind of hardware would you need?
05:22What is the inference engine of choice uh, for for all of this?
05:27So how did we do this?
05:28Uh, we really wanted to compare at a task level, so we looked at uh, long horizon tasks specifically because long horizon tasks are really the perfect uh, um, if you really have your own infrastructure that you want to uh, to own.
05:41If you do do it very very interactive, it might not be the economies might not be the same, but for long horizon tasks, we think that that's the first reason or the first pivot point where um, taking
05:52a hold of your AI stack is really relevant.
05:55So we took one specific one being um, uh, the SWE-bench pro uh, tasks, which is set of benchmark tasks that exist um, uh, for coding assistance.
06:06And what we did is we kind of validate or kind of very different types of coding harnesses.
06:11We took Codex by and and Cloud Code for this, which are more or less the three most popular harnesses out there for for coding assistance.
06:19And then we kind of vary the different types of models that we could use on this as a as well.
06:24Starting with a traditional standard API environment, [clears throat] but then also looking at a wide range of different open weight models with different types of complexities and and parameter sizes as a as well.
06:39But as this is a talk about translate translating this towards your own infrastructure, we of course also map this to the right hardware architecture at hand as well.
06:48Always trying to find the right type architecture that kind of is still sufficient enough to really do this at at at scale and it where you really have where you can have multiple concurrency
06:58concurrent users on on this.
07:01So in essence, we kind of increased the hardware complexity, increased the model complexity, but at the same time also looked at really the application performance of all of this.
07:09So what is the kind of the resolve rate of your task?
07:12What is the latency and what is the cost?
07:14All in all in all this was more than a thousands of different runs that we did.
07:19We set up a full benchmarking pipeline and I kind of want to run you run you down with the different results we got out of out of this.
07:28So let's start maybe with the most obvious one.
07:30Kind of different models, different model complexities, different model parameters sizes actually give you different accuracies of these of these tasks.
07:41We could actually really classify three main categories of of models.
07:47The more smaller ones and with smaller ones I would say up to a few hundred billion of parameters where Quen and and DeepSeeker are part of.
07:57They kind of got a got into a resolve rate of about 32 5 to 40% of tasks that really were able to achieve accurately correctly.
08:08Second one like GLM, um, was really at Opus level, uh, uh, range and then Kimik 3 that came out a few months ago, um, was, uh, really was really crushing it to,
08:20in all honesty, also probably because part of it has been probably in the training set as well, but still the the the results were quite, uh, substantial.
08:28Had a resolve rate of 86%.
08:30Now, all of this is not something that uh, is is something that you can easily find on artificial analysis as well.
08:36This doesn't really give you a lot of uh, uh, additional insights.
08:40What I think is really more interesting is if you add the infrastructure and hardware angle to it as well.
08:45And there we do see quite a bit of of of differences.
08:48What I think is interesting is if you compare for example Qwen with with DeepSeek that all in all the application performance of it is more or less on the same,
08:58uh, level.
08:59Although if you then look at what kind of hardware you would need for Qwen and DeepSeek, you're looking at a factor of four and in terms of renting GPUs that are is more expensive.
09:08Um, similarly if you just make the move from, um, a GLM model to, uh, a Kimik 3, uh, model, it gives you a bump of about 20% in in accuracy, but all in all it's just giving you only a 15%
09:21increase in cost in renting GPUs as well.
09:24So what does that tell you?
09:25It basically tells you that if you move this make this move towards open weight models that it's super super important to kind of look at the hardware constraints of this as well and also look look at the actual,
09:38um, costs of, uh, of all of this and there's definitely not a linear curve to be found here.
09:45So often it's just a step wise function where you where it might be that you're choosing a specific, uh, model and that just taking a slightly better or slightly,
09:55uh, less performant model would have a huge impact on the hardware architecture of choice as as well.
10:02So that's one and that's looking at mainly from an application performance perspective.
10:06Um, let's take a different angle to it and that's concurrency.
10:10Um because when you're using an API, um you can um get a certain performance of of token consumption.
10:19You know, you know more or less what your token latency is going to be, and you know what your token's throughput is going to be.
10:24The moment you make the move towards a more infrastructure-based solution, suddenly you need to worry about concurrency as well.
10:31In other words, how many users can I reliably uh put on the same uh machine without the system completely deteriorating from a latency perspective in that that sense.
10:42There's a a memory size to it, but there's also a latency uh perspective to it.
10:47Uh memory means that uh on the one hand, the most systems are memory bound, and it means that on the one hand, your model parameter size need to fit your your your memory, but also your user's KV cache as a as well.
11:00And as you have an increasing number of um of users uh using this the system, your KV cache uh requirement will become bigger and bigger as a as well.
11:10So, what we did is we kind of tried to look at the different types of solutions I just already showed you, and look at how many concurrent users we were actually able to fit really comfortably
11:19in that uh that system.
11:21And the results were pretty um uh surprising, or at least also maybe not that positive for really the bigger models as a as as well.
11:31So, bear in mind, each of these different models have their own hardware architecture that is uh that is optimal for it.
11:36But if you then look at the comfortable concurrency levels that we were able to get, it really really dropped down very significantly, even with bigger hardware architectures as um as well.
11:47And in all cases, we were much slower than uh uh our baseline, which was uh Cloud Code in that that sense.
11:55By the way, I think you figured it all out already.
11:57Um we actually have a detailed blog post on each of the topics I'm I'm talking about.
12:01Every QR code actually leads you to to that as um as as well.
12:06So, this kind of shows you that uh concurrency is actually probably even more important than accuracy in the in this sense.
12:13Accuracy gives you the kind of baseline, but then even for the really big models and even if especially if you look at very long horizon tasks like code assistance,
12:23running a Kimi K3 model only gives you about four concurrent users that can really run reliably on a on this and still the latency is still a factor three slower.
12:37So, what we then did is a kind of full sweet spots analysis of okay, in which cases is it really interesting to stick to your API usage?
12:46In which cases would it make sense more for you to kind of move to renting out GPUs with the the various model providers that are out there and in which case would you really
12:58go for would it be more interesting for purchasing and really investing in your own infrastructure and building it out on prem.
13:06Of course, there can be a lot of other reasons as well to make that that switch but if we look at it from a total cost of ownership perspective.
13:14So, without going into too much details what we saw there is that the lower your hardware architecture system actually is, the worse it is to move from API to your own infrastructure.
13:25We saw that if you really are at the lower tiers of of of models and as a result also at the lower tiers of the hardware architecture that you would need,
13:33you actually need an GPU utilization of about 89% so that means that your GPU needs to run at capacity at 89% of the of the time.
13:45Um, similarly if you actually move to more bigger hardware systems and I made a mistake there it should be a higher hardware systems.
13:52So, for example an 8 times B200s, you're okay with just 15% of of GPU utilization and that's really realistic and 89% is just completely infeasible in that that sense.
14:05If you then kind of average it out and if you look at for example GLM class applications, what we do see is that that's really the sweet spot or the threshold of when
14:15moving from API to infrastructure makes it more interesting as a as well.
14:20And in the full total cost of ownership analysis that we did specifically for this application being coding assistance, we really saw that you get into a kind of 24 5%
14:30of of cost saving.
14:33So that's really infrastructure perspective.
14:34Let's kind of switch it over to harnesses.
14:37We also varied the amount the harnesses as a as as well and looked at what the application performance was for those three different harnesses.
14:45And on the first glance, the application performance actually didn't really change that much, which was maybe kind of surprising.
14:52Every single task we we gave every single harness we gave you more or less got it and every single for different kind of models, we kind of got into the same kind of application accuracy on this
15:03on this benchmark.
15:03So at the first glance, you could say, well, the harness actually doesn't have an impact whatsoever on that because it gets the job done.
15:11Well, that might be the case if you're going for an API only usage, but it's completely different if you're moving towards your own infrastructure and you actually pay for the GPU
15:21usage because then the wall time of your GPU really is is relevant.
15:25And what we saw there is that different kinds of harnesses actually had a huge impact on the amount of token consumption we had in the in this and then specifically
15:35in the case of of Codex, we had for a GLM only half of the of tokens that were actually being being generated.
15:44And as a result, also the wall time of using these GPUs actually was half of that as well.
15:50So that means that although the harnesses actually gave you the right type of application performance or more or less a similar level of application performance, the GPU bill you would in the end have would be half
16:03of of the other harnesses as well.
16:06So it really shows very clearly that in that case also the right harness to model choice might not have an impact on application performance, but it has really an impact
16:15on your total cost of ownership for your GPUs as a as well.
16:20So, this is just the first study that we're that we did in the last few few months on this.
16:27Where are we going with this?
16:27I think there's a lot to unpack.
16:31It's very clear that moving this from API to infrastructure that is just not very simple or very linear choice.
16:38And I have showed this now for one single application being coding assistance, but a real enterprise mix would often have a combination of a lot of difference of applications
16:49as as as well.
16:50And that actually complicates things even even further.
16:54We also saw that there's a very complex relationship between harnesses, between the models, between the infrastructure.
16:59And I haven't touched upon other things like what do we do in terms of quantization?
17:04What is the impact of specific optimizations like speculative decoding that we applied as well, but that we haven't configured into to detail.
17:13And the third dimension is of course that it's very much a constantly moving target as well.
17:17New models are being being launched.
17:20New hardware architecture choices are there as as well.
17:24So, we're actually building out a kind of benchmarking pipeline that we already have.
17:30We're building this out to a full benchmarking data set as well that we're going to share publicly for for everyone and trying to keep track of this moving target
17:40moving moving forward.
17:42But at the same time we cannot do this alone.
17:45We're a research organization.
17:46We're a non-profit research organization and the biggest important part for us is trying to bring that ecosystem together.
17:53And I think that the software world and the hardware world deserves to be much closer together as well.
18:00So, we're actually launching a kind of alliance to bring people who are interested in this part, software people from the software who are interested in hardware optimization.
18:09We're bringing this together with our existing customer portfolio of new and existing chip providers and infrastructure providers as well.
18:17And if you would be interested in this, uh free to reach out, uh after this uh this talk.
18:23Thanks a lot.