WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=9-Y_oS9OBLc NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/9-Y_oS9OBLc.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:12.560 --> 00:00:13.640 So my name is Steven. s2 00:00:13.640 --> 00:00:17.880 I'm leading the AI department at IMEC. s3 00:00:17.880 --> 00:00:24.880 For those of you who don't know what IMEC is, we're not that well known actually in the AI space, but we're very well known known in the semiconductor space. s4 00:00:24.880 --> 00:00:27.520 Been seen a little bit as the the hidden gem. s5 00:00:27.520 --> 00:00:36.200 We exist already for about more than four decades and what we typically do is we have the most advanced R&D clean room in the world and we actually run s6 00:00:36.200 --> 00:00:40.280 together with our partners next generation semiconductors in that sense. s7 00:00:40.280 --> 00:00:44.760 And so actually the impact that we've had in that semiconductor space is pretty huge. s8 00:00:44.760 --> 00:00:54.120 I think every single transistor technology that was more or less that is deployed now has been invented by us in the last four four decades in that that sense. s9 00:00:54.120 --> 00:00:58.960 So in that sense pretty much a hidden gem in in in Belgium where I'm where I'm from. s10 00:00:58.960 --> 00:01:04.040 So why is a a semiconductor nerd talking about artificial intelligence? s11 00:01:04.040 --> 00:01:06.680 Well, we do way more than just transistor technology. s12 00:01:06.680 --> 00:01:08.720 We go all the way up to systems. s13 00:01:08.720 --> 00:01:18.320 We go all the way up to the workloads as as well and that's where I'm responsible from for where we're trying to connect what AI workloads does to infrastructure s14 00:01:18.320 --> 00:01:20.240 in that that sense. s15 00:01:20.240 --> 00:01:24.360 On the one hand because we've lived that as well the last few few years. s16 00:01:24.360 --> 00:01:32.320 Of course within IMEC we're also working very much with with AI to kind of realize agentica flows as as well. s17 00:01:32.320 --> 00:01:39.520 And what we've experienced I think a lot of you of us have experienced at the last few few years is kind of a different stages of AI maturity. s18 00:01:39.520 --> 00:01:52.600 We we kind of started off in 2024 2025 thinking that we needed the latest and the greatest of every single frontier model and that the best wasn't good enough and that we needed more and more. s19 00:01:52.600 --> 00:01:57.160 But of course at the same time what we realized is that prices started evolve. s20 00:01:57.160 --> 00:02:03.692 We moved from more subscription based pricing to more token-based uh pricing. s21 00:02:03.692 --> 00:02:14.252 And then like the last year, we moved more and more towards model routing where we try to find the best type of model for the task at at hand. s22 00:02:14.252 --> 00:02:20.812 Things that I think all of us have been have been doing the last year or so as as as well. s23 00:02:20.812 --> 00:02:29.772 What we're actually seeing as well with both within the company but also with the customers that we're that we're talking about is that there's other stages to that as as well. s24 00:02:29.772 --> 00:02:38.812 Model routing is in that sense maybe the first just low-hanging fruits in in getting a good optimization of finding the right model to the right task. s25 00:02:38.812 --> 00:02:51.212 We're seeing more and more as as token consumption really really grows that there's a need for doing other things as well and doing other things basically means taking control of your own stack in that that cell. s26 00:02:51.212 --> 00:02:52.172 That sense. s27 00:02:52.172 --> 00:03:01.892 So we're seeing customers and we're as on that path as well moving from well, first of all, we started with just looking at closed APIs, just grabbing what we could do. s28 00:03:01.892 --> 00:03:07.012 Then second phase was for us more, okay, let's see how we can do cost optimization. s29 00:03:07.012 --> 00:03:18.772 We're very much ourself now in the the third phase where we're looking at how can we host open weights models and how can we lease our infrastructure in the long run maybe even s30 00:03:18.772 --> 00:03:22.172 buy our infrastructure as as well. s31 00:03:22.172 --> 00:03:32.732 I particularly want to talk about this third and fourth phase of of of all of this and see what it actually would would mean because running an API or just having an API s32 00:03:32.732 --> 00:03:36.012 for to for token consumption is pretty damn straightforward. s33 00:03:36.012 --> 00:03:45.012 It's pretty easy to to do but the moment you move towards this third and fourth phase, suddenly the design space starts blowing up as as well. s34 00:03:45.012 --> 00:03:53.252 In the first phase, we're just worried about what is the right type of model for the right type of model accuracy or for the right type of task complexity. s35 00:03:53.252 --> 00:04:00.932 But the moment you start owning your your infrastructure, you suddenly need to worry about a whole lot of other things as as well. s36 00:04:00.932 --> 00:04:02.694 What kind of model am I going to take? s37 00:04:02.694 --> 00:04:04.854 How can I trust this this model? s38 00:04:04.854 --> 00:04:15.534 Uh, you need to think about harnesses, but you also need to think about a lot of hardware configurations as as as well, such as what is the right hardware architecture that I would would use for for this? s39 00:04:15.534 --> 00:04:17.374 What is the CPU to GPU balance? s40 00:04:17.374 --> 00:04:22.134 How can I do KB cache optimization and uh, and all of that. s41 00:04:22.134 --> 00:04:33.974 And so that's where we want to position ourselves as I am going forward as well, where of course here in a in a world that is very much aware of models and harnesses, but we do see very much also a gap from those models and harnesses s42 00:04:33.974 --> 00:04:37.254 to that infrastructure as as as well. s43 00:04:37.254 --> 00:04:44.414 Um, so what we did in the last few months and what I'm here to present today is a kind of benchmarking study that we did in the last few months. s44 00:04:44.414 --> 00:04:55.574 We did uh, really thousands of of runs to really really quantify what does that move from going from an API to really infrastructure really uh, give you. s45 00:04:55.574 --> 00:04:57.854 And so really from an Agentic AI perspective. s46 00:04:57.854 --> 00:05:05.934 So if you have an Agentic AI task, if you have a specific Agentic AI task performance, what would be the cost for you if you move from an API s47 00:05:05.934 --> 00:05:08.654 towards your own uh, infrastructure? s48 00:05:08.654 --> 00:05:14.894 So that also means moving from a more token-based cost subscription to really kind of total cost of of ownership. s49 00:05:14.894 --> 00:05:18.134 And also um, how should you then configure the infrastructure? s50 00:05:18.134 --> 00:05:20.894 Really from what kind of harnesses are relevant? s51 00:05:20.894 --> 00:05:22.374 Uh, what kind of hardware would you need? s52 00:05:22.374 --> 00:05:27.134 What is the inference engine of choice uh, for for all of this? s53 00:05:27.134 --> 00:05:28.094 So how did we do this? s54 00:05:28.094 --> 00:05:41.494 Uh, we really wanted to compare at a task level, so we looked at uh, long horizon tasks specifically because long horizon tasks are really the perfect uh, um, if you really have your own infrastructure that you want to uh, to own. s55 00:05:41.494 --> 00:05:52.974 If you do do it very very interactive, it might not be the economies might not be the same, but for long horizon tasks, we think that that's the first reason or the first pivot point where um, taking s56 00:05:52.974 --> 00:05:55.974 a hold of your AI stack is really relevant. s57 00:05:55.974 --> 00:06:06.335 So we took one specific one being um, uh, the SWE-bench pro uh, tasks, which is set of benchmark tasks that exist um, uh, for coding assistance. s58 00:06:06.335 --> 00:06:11.255 And what we did is we kind of validate or kind of very different types of coding harnesses. s59 00:06:11.255 --> 00:06:19.695 We took Codex by and and Cloud Code for this, which are more or less the three most popular harnesses out there for for coding assistance. s60 00:06:19.695 --> 00:06:24.415 And then we kind of vary the different types of models that we could use on this as a as well. s61 00:06:24.415 --> 00:06:39.055 Starting with a traditional standard API environment, [clears throat] but then also looking at a wide range of different open weight models with different types of complexities and and parameter sizes as a as well. s62 00:06:39.055 --> 00:06:48.015 But as this is a talk about translate translating this towards your own infrastructure, we of course also map this to the right hardware architecture at hand as well. s63 00:06:48.015 --> 00:06:58.655 Always trying to find the right type architecture that kind of is still sufficient enough to really do this at at at scale and it where you really have where you can have multiple concurrency s64 00:06:58.655 --> 00:07:01.175 concurrent users on on this. s65 00:07:01.175 --> 00:07:09.775 So in essence, we kind of increased the hardware complexity, increased the model complexity, but at the same time also looked at really the application performance of all of this. s66 00:07:09.775 --> 00:07:12.455 So what is the kind of the resolve rate of your task? s67 00:07:12.455 --> 00:07:14.495 What is the latency and what is the cost? s68 00:07:14.495 --> 00:07:19.015 All in all in all this was more than a thousands of different runs that we did. s69 00:07:19.015 --> 00:07:28.215 We set up a full benchmarking pipeline and I kind of want to run you run you down with the different results we got out of out of this. s70 00:07:28.215 --> 00:07:30.895 So let's start maybe with the most obvious one. s71 00:07:30.895 --> 00:07:41.695 Kind of different models, different model complexities, different model parameters sizes actually give you different accuracies of these of these tasks. s72 00:07:41.695 --> 00:07:47.415 We could actually really classify three main categories of of models. s73 00:07:47.415 --> 00:07:57.215 The more smaller ones and with smaller ones I would say up to a few hundred billion of parameters where Quen and and DeepSeeker are part of. s74 00:07:57.215 --> 00:08:08.215 They kind of got a got into a resolve rate of about 32 5 to 40% of tasks that really were able to achieve accurately correctly. s75 00:08:08.215 --> 00:08:20.137 Second one like GLM, um, was really at Opus level, uh, uh, range and then Kimik 3 that came out a few months ago, um, was, uh, really was really crushing it to, s76 00:08:20.137 --> 00:08:28.657 in all honesty, also probably because part of it has been probably in the training set as well, but still the the the results were quite, uh, substantial. s77 00:08:28.657 --> 00:08:30.497 Had a resolve rate of 86%. s78 00:08:30.497 --> 00:08:36.617 Now, all of this is not something that uh, is is something that you can easily find on artificial analysis as well. s79 00:08:36.617 --> 00:08:40.577 This doesn't really give you a lot of uh, uh, additional insights. s80 00:08:40.577 --> 00:08:45.577 What I think is really more interesting is if you add the infrastructure and hardware angle to it as well. s81 00:08:45.577 --> 00:08:48.177 And there we do see quite a bit of of of differences. s82 00:08:48.177 --> 00:08:58.057 What I think is interesting is if you compare for example Qwen with with DeepSeek that all in all the application performance of it is more or less on the same, s83 00:08:58.057 --> 00:08:59.017 uh, level. s84 00:08:59.017 --> 00:09:08.497 Although if you then look at what kind of hardware you would need for Qwen and DeepSeek, you're looking at a factor of four and in terms of renting GPUs that are is more expensive. s85 00:09:08.497 --> 00:09:21.937 Um, similarly if you just make the move from, um, a GLM model to, uh, a Kimik 3, uh, model, it gives you a bump of about 20% in in accuracy, but all in all it's just giving you only a 15% s86 00:09:21.937 --> 00:09:24.577 increase in cost in renting GPUs as well. s87 00:09:24.577 --> 00:09:25.777 So what does that tell you? s88 00:09:25.777 --> 00:09:38.937 It basically tells you that if you move this make this move towards open weight models that it's super super important to kind of look at the hardware constraints of this as well and also look look at the actual, s89 00:09:38.937 --> 00:09:45.217 um, costs of, uh, of all of this and there's definitely not a linear curve to be found here. s90 00:09:45.217 --> 00:09:55.137 So often it's just a step wise function where you where it might be that you're choosing a specific, uh, model and that just taking a slightly better or slightly, s91 00:09:55.137 --> 00:10:02.337 uh, less performant model would have a huge impact on the hardware architecture of choice as as well. s92 00:10:02.337 --> 00:10:06.577 So that's one and that's looking at mainly from an application performance perspective. s93 00:10:06.577 --> 00:10:10.658 Um, let's take a different angle to it and that's concurrency. s94 00:10:10.658 --> 00:10:19.138 Um because when you're using an API, um you can um get a certain performance of of token consumption. s95 00:10:19.138 --> 00:10:24.978 You know, you know more or less what your token latency is going to be, and you know what your token's throughput is going to be. s96 00:10:24.978 --> 00:10:31.098 The moment you make the move towards a more infrastructure-based solution, suddenly you need to worry about concurrency as well. s97 00:10:31.098 --> 00:10:42.898 In other words, how many users can I reliably uh put on the same uh machine without the system completely deteriorating from a latency perspective in that that sense. s98 00:10:42.898 --> 00:10:47.378 There's a a memory size to it, but there's also a latency uh perspective to it. s99 00:10:47.378 --> 00:11:00.258 Uh memory means that uh on the one hand, the most systems are memory bound, and it means that on the one hand, your model parameter size need to fit your your your memory, but also your user's KV cache as a as well. s100 00:11:00.258 --> 00:11:10.298 And as you have an increasing number of um of users uh using this the system, your KV cache uh requirement will become bigger and bigger as a as well. s101 00:11:10.298 --> 00:11:19.978 So, what we did is we kind of tried to look at the different types of solutions I just already showed you, and look at how many concurrent users we were actually able to fit really comfortably s102 00:11:19.978 --> 00:11:21.698 in that uh that system. s103 00:11:21.698 --> 00:11:31.098 And the results were pretty um uh surprising, or at least also maybe not that positive for really the bigger models as a as as well. s104 00:11:31.098 --> 00:11:36.858 So, bear in mind, each of these different models have their own hardware architecture that is uh that is optimal for it. s105 00:11:36.858 --> 00:11:47.778 But if you then look at the comfortable concurrency levels that we were able to get, it really really dropped down very significantly, even with bigger hardware architectures as um as well. s106 00:11:47.778 --> 00:11:55.418 And in all cases, we were much slower than uh uh our baseline, which was uh Cloud Code in that that sense. s107 00:11:55.418 --> 00:11:57.458 By the way, I think you figured it all out already. s108 00:11:57.458 --> 00:12:01.738 Um we actually have a detailed blog post on each of the topics I'm I'm talking about. s109 00:12:01.738 --> 00:12:06.618 Every QR code actually leads you to to that as um as as well. s110 00:12:06.618 --> 00:12:13.299 So, this kind of shows you that uh concurrency is actually probably even more important than accuracy in the in this sense. s111 00:12:13.299 --> 00:12:23.699 Accuracy gives you the kind of baseline, but then even for the really big models and even if especially if you look at very long horizon tasks like code assistance, s112 00:12:23.699 --> 00:12:36.699 running a Kimi K3 model only gives you about four concurrent users that can really run reliably on a on this and still the latency is still a factor three slower. s113 00:12:37.739 --> 00:12:46.779 So, what we then did is a kind of full sweet spots analysis of okay, in which cases is it really interesting to stick to your API usage? s114 00:12:46.779 --> 00:12:58.619 In which cases would it make sense more for you to kind of move to renting out GPUs with the the various model providers that are out there and in which case would you really s115 00:12:58.619 --> 00:13:06.219 go for would it be more interesting for purchasing and really investing in your own infrastructure and building it out on prem. s116 00:13:06.219 --> 00:13:14.499 Of course, there can be a lot of other reasons as well to make that that switch but if we look at it from a total cost of ownership perspective. s117 00:13:14.499 --> 00:13:25.099 So, without going into too much details what we saw there is that the lower your hardware architecture system actually is, the worse it is to move from API to your own infrastructure. s118 00:13:25.099 --> 00:13:33.819 We saw that if you really are at the lower tiers of of of models and as a result also at the lower tiers of the hardware architecture that you would need, s119 00:13:33.819 --> 00:13:45.499 you actually need an GPU utilization of about 89% so that means that your GPU needs to run at capacity at 89% of the of the time. s120 00:13:45.499 --> 00:13:52.819 Um, similarly if you actually move to more bigger hardware systems and I made a mistake there it should be a higher hardware systems. s121 00:13:52.819 --> 00:14:05.819 So, for example an 8 times B200s, you're okay with just 15% of of GPU utilization and that's really realistic and 89% is just completely infeasible in that that sense. s122 00:14:05.819 --> 00:14:15.421 If you then kind of average it out and if you look at for example GLM class applications, what we do see is that that's really the sweet spot or the threshold of when s123 00:14:15.421 --> 00:14:20.061 moving from API to infrastructure makes it more interesting as a as well. s124 00:14:20.061 --> 00:14:30.381 And in the full total cost of ownership analysis that we did specifically for this application being coding assistance, we really saw that you get into a kind of 24 5% s125 00:14:30.381 --> 00:14:33.261 of of cost saving. s126 00:14:33.381 --> 00:14:34.941 So that's really infrastructure perspective. s127 00:14:34.941 --> 00:14:37.101 Let's kind of switch it over to harnesses. s128 00:14:37.101 --> 00:14:45.741 We also varied the amount the harnesses as a as as well and looked at what the application performance was for those three different harnesses. s129 00:14:45.741 --> 00:14:52.501 And on the first glance, the application performance actually didn't really change that much, which was maybe kind of surprising. s130 00:14:52.501 --> 00:15:03.261 Every single task we we gave every single harness we gave you more or less got it and every single for different kind of models, we kind of got into the same kind of application accuracy on this s131 00:15:03.261 --> 00:15:03.981 on this benchmark. s132 00:15:03.981 --> 00:15:11.501 So at the first glance, you could say, well, the harness actually doesn't have an impact whatsoever on that because it gets the job done. s133 00:15:11.501 --> 00:15:21.221 Well, that might be the case if you're going for an API only usage, but it's completely different if you're moving towards your own infrastructure and you actually pay for the GPU s134 00:15:21.221 --> 00:15:25.861 usage because then the wall time of your GPU really is is relevant. s135 00:15:25.861 --> 00:15:35.421 And what we saw there is that different kinds of harnesses actually had a huge impact on the amount of token consumption we had in the in this and then specifically s136 00:15:35.421 --> 00:15:44.981 in the case of of Codex, we had for a GLM only half of the of tokens that were actually being being generated. s137 00:15:44.981 --> 00:15:50.621 And as a result, also the wall time of using these GPUs actually was half of that as well. s138 00:15:50.621 --> 00:16:03.581 So that means that although the harnesses actually gave you the right type of application performance or more or less a similar level of application performance, the GPU bill you would in the end have would be half s139 00:16:03.581 --> 00:16:06.341 of of the other harnesses as well. s140 00:16:06.341 --> 00:16:15.821 So it really shows very clearly that in that case also the right harness to model choice might not have an impact on application performance, but it has really an impact s141 00:16:15.821 --> 00:16:20.742 on your total cost of ownership for your GPUs as a as well. s142 00:16:20.742 --> 00:16:27.022 So, this is just the first study that we're that we did in the last few few months on this. s143 00:16:27.022 --> 00:16:27.982 Where are we going with this? s144 00:16:27.982 --> 00:16:31.102 I think there's a lot to unpack. s145 00:16:31.102 --> 00:16:38.222 It's very clear that moving this from API to infrastructure that is just not very simple or very linear choice. s146 00:16:38.222 --> 00:16:49.262 And I have showed this now for one single application being coding assistance, but a real enterprise mix would often have a combination of a lot of difference of applications s147 00:16:49.262 --> 00:16:50.862 as as as well. s148 00:16:50.862 --> 00:16:54.702 And that actually complicates things even even further. s149 00:16:54.702 --> 00:16:59.462 We also saw that there's a very complex relationship between harnesses, between the models, between the infrastructure. s150 00:16:59.462 --> 00:17:04.022 And I haven't touched upon other things like what do we do in terms of quantization? s151 00:17:04.022 --> 00:17:13.022 What is the impact of specific optimizations like speculative decoding that we applied as well, but that we haven't configured into to detail. s152 00:17:13.022 --> 00:17:17.182 And the third dimension is of course that it's very much a constantly moving target as well. s153 00:17:17.182 --> 00:17:20.782 New models are being being launched. s154 00:17:20.782 --> 00:17:24.262 New hardware architecture choices are there as as well. s155 00:17:24.262 --> 00:17:30.902 So, we're actually building out a kind of benchmarking pipeline that we already have. s156 00:17:30.902 --> 00:17:40.582 We're building this out to a full benchmarking data set as well that we're going to share publicly for for everyone and trying to keep track of this moving target s157 00:17:40.582 --> 00:17:42.662 moving moving forward. s158 00:17:42.662 --> 00:17:45.222 But at the same time we cannot do this alone. s159 00:17:45.222 --> 00:17:46.542 We're a research organization. s160 00:17:46.542 --> 00:17:53.662 We're a non-profit research organization and the biggest important part for us is trying to bring that ecosystem together. s161 00:17:53.662 --> 00:18:00.302 And I think that the software world and the hardware world deserves to be much closer together as well. s162 00:18:00.302 --> 00:18:09.342 So, we're actually launching a kind of alliance to bring people who are interested in this part, software people from the software who are interested in hardware optimization. s163 00:18:09.342 --> 00:18:17.822 We're bringing this together with our existing customer portfolio of new and existing chip providers and infrastructure providers as well. s164 00:18:17.822 --> 00:18:23.343 And if you would be interested in this, uh free to reach out, uh after this uh this talk. s165 00:18:23.343 --> 00:18:25.583 Thanks a lot.