WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=Ndp5VaqFrl0 NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/Ndp5VaqFrl0.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:15.080 --> 00:00:16.400 Hello, hello. s2 00:00:16.400 --> 00:00:18.000 Um, I guess we're starting. s3 00:00:18.000 --> 00:00:26.240 I know some of you are still joining in, but I'll go on and you can jump in at any time. s4 00:00:26.240 --> 00:00:30.080 Uh, so you might have seen the title of this talk. s5 00:00:30.080 --> 00:00:39.600 Um, and what I have on the slides here is slightly different, but I promise it's the same direction and the same general idea. s6 00:00:39.600 --> 00:00:49.880 So, it all started for me wanting to build a use case that's around privacy and token costs, basically. s7 00:00:49.880 --> 00:00:52.320 So, um, I started with Obsidian. s8 00:00:52.320 --> 00:00:54.760 I had some markdown files there. s9 00:00:54.760 --> 00:01:02.560 Um, and as you know, it's like you have this nice, um, graphs and it's all good. s10 00:01:02.560 --> 00:01:04.640 Looks very nice. s11 00:01:04.640 --> 00:01:12.080 However, um, I had certain issues beyond just organizing the files. s12 00:01:12.080 --> 00:01:20.000 So, a lot of people do this knowledge graphs and so on and like memory for agents and that's quite good. s13 00:01:20.000 --> 00:01:27.160 However, I needed to approach this differently because there was a concern around privacy. s14 00:01:27.160 --> 00:01:42.040 So, when using agents, when using cloud code, when using co-work, uh, you can expose your files and for some institutions, for some people, this was, um, a requirement that, s15 00:01:42.040 --> 00:01:44.200 um, needed to be met. s16 00:01:44.200 --> 00:01:56.360 So, the goal was to keep things private and self-hosted, but also make them quite usable and friendly so that people could actually do these things. s17 00:01:56.360 --> 00:02:13.400 And, um, there was kind of this openness to fine-tune, to be very adaptable, to use open-source models, and, um I found that the solution was, um in Superlinked, which is s18 00:02:13.400 --> 00:02:17.320 a model inference, um engine that's open-source. s19 00:02:17.320 --> 00:02:21.400 So, it has both a cluster and a model inference solution. s20 00:02:21.400 --> 00:02:35.360 And, I will be talking about that today, and how I managed to kind of merge these concepts together, and, um create something that leans on these three things. s21 00:02:35.360 --> 00:02:43.840 So, the first main idea was I wanted to make Claude Code a bit cheaper and save on tokens. s22 00:02:43.840 --> 00:02:59.440 So, we know how with the release of Fable, there initially there was this, um API-only pricing, and that kind of, uh maybe disappointed a few people, or the token economics got a bit more attention now. s23 00:02:59.440 --> 00:03:00.880 So, that was one thing. s24 00:03:00.880 --> 00:03:11.840 Especially for document processing, sensitive document processing, where you need to do OCR, there are a lot of scans, images, and stuff like this, the token cost can ramp up. s25 00:03:11.840 --> 00:03:26.600 So, I wanted to find a way to kind of do this, but also having smaller artifacts means that you get better context, so you kind of limit context loss in a way, and you don't like s26 00:03:26.600 --> 00:03:34.560 fill out, um the context with data that can be processed in some way else. s27 00:03:34.560 --> 00:03:37.600 Um, and I wanted to make this private. s28 00:03:37.600 --> 00:03:45.000 So, yes, for certain situations, it's fine to paste everything into Claude Code or have that API working there. s29 00:03:45.000 --> 00:03:54.920 But, what if you could kind of self-host and have your own VPC or your own cloud to preprocess your sensitive documents and then return something back. s30 00:03:54.920 --> 00:03:59.600 So, this was another target of mine when building this. s31 00:03:59.600 --> 00:04:14.959 And also I wanted adaptability, so it wasn't just um privacy and lowering token costs, but also having the chance to test out multiple models, maybe host Laura's that are very task specific s32 00:04:14.959 --> 00:04:24.840 because with this very sensitive use cases, usually there is some sort of fine-tuning or adaptation towards that niche use case. s33 00:04:24.840 --> 00:04:28.680 So, I had to do that as well. s34 00:04:28.680 --> 00:04:47.000 And the setup um involves Superlinked um and Sci the repo, so you can for the purpose of today, you can go to um the repo which is fully open source, so you can find it on GitHub. s35 00:04:47.000 --> 00:04:49.240 It's Superlinked Sci. s36 00:04:49.240 --> 00:04:51.760 Uh you can Yeah. s37 00:04:51.760 --> 00:04:52.560 So, this is it. s38 00:04:52.560 --> 00:04:53.880 This is the logo. s39 00:04:53.880 --> 00:05:01.120 And I'll be talking about um an integration that's uh Sorry, not here. s40 00:05:01.120 --> 00:05:07.919 So, an integra an MCP integration which is basically a package that we did, which is this one here. s41 00:05:07.919 --> 00:05:12.560 So, you can also have a look at this subdirectory. s42 00:05:12.560 --> 00:05:21.640 So, this is where most of the talk will be about, but I'll also mention why actually the inference was the bottleneck when building something like this. s43 00:05:21.640 --> 00:05:30.760 So, it was easy to figure out how to store the data, maybe in Obsidian and so on and have these connections between the data. s44 00:05:30.760 --> 00:05:34.440 Um That was the easy part. s45 00:05:34.440 --> 00:05:35.960 I could also use an API. s46 00:05:35.960 --> 00:05:47.440 However, if I wanted to do open source, if I wanted to do open source inference at scale and testing out multiple models, being able to deploy this in production, s47 00:05:47.440 --> 00:06:01.360 things got a bit difficult and that's why the infrastructure advantage that Sci offers, the fact that you're able to kind of deploy both a cluster and the models themselves s48 00:06:01.360 --> 00:06:05.840 was a very big advantage in my case. s49 00:06:05.840 --> 00:06:10.360 Um here you can see this um Obsidian example. s50 00:06:10.360 --> 00:06:13.280 So I'll work with kind of some of these things. s51 00:06:13.280 --> 00:06:16.240 Uh but you can also try this yourself. s52 00:06:16.240 --> 00:06:21.800 Um however, I will be using a managed cluster. s53 00:06:21.800 --> 00:06:26.360 We cannot um I cannot give that to you right now live. s54 00:06:26.360 --> 00:06:34.680 However, you can go to our website, which is this one, and here you have this apply for in inference grant. s55 00:06:34.680 --> 00:06:41.600 So you can go there and you'll receive your um API key that you can try and you can do things quicker. s56 00:06:41.600 --> 00:06:45.919 If not, you can do things locally cuz there is a Docker container and all that. s57 00:06:45.919 --> 00:06:55.760 Uh it will just take a bit slower to do these things, but I'd prefer if you kind of listen to me now and then try things after, but also in parallel if you want. s58 00:06:55.760 --> 00:07:07.120 And we also have this very cool um kind of QR codes memorabilia stuff where you can get an even better cluster, as I think, and you can find them here s59 00:07:07.120 --> 00:07:10.160 uh after the talk. s60 00:07:10.560 --> 00:07:19.919 So, um as I said, the idea was to have this privacy element and have a way to offload. s61 00:07:19.919 --> 00:07:24.200 So basically, I didn't want to disrupt my normal usage day-to-day. s62 00:07:24.200 --> 00:07:37.560 Um so I'm just working in cloud code and then if there's a specific task that needs to be done on my virtual private clouds that needs to be processed, that's sensitive, that costs a lot of tokens, s63 00:07:37.560 --> 00:07:40.360 I can just offload it and continue my work. s64 00:07:40.360 --> 00:07:47.320 So, the natural direction was to build some sort of um skill or plugin. s65 00:07:47.320 --> 00:08:01.120 But, this was even elevated more by building a whole MCP server that's that gets called in cloud code and there's that element of like using skills, but also having s66 00:08:01.120 --> 00:08:07.000 actually a dedicated edge that can actually do the heavy lifting. s67 00:08:07.000 --> 00:08:22.240 And you can also benefit from Superlinks inference, which I will talk about later on kind of how we've done this and what are the advantages of using um Superlinks inference engine. s68 00:08:22.320 --> 00:08:33.320 So, it saves tokens because for example, a number would be to have like maybe 10 pages of a PDF that has images, different details in there. s69 00:08:33.320 --> 00:08:47.560 So, if you pre-process that with good open-source OCR models and then return to your main agent harness or cloud code or whatever you're using, you get the context, but also s70 00:08:47.560 --> 00:08:52.640 benefit on the lower token spent and better context and so on. s71 00:08:52.640 --> 00:08:58.560 So, that's the general idea, especially for OCR this is apparent. s72 00:08:58.560 --> 00:09:07.120 However, there are multiple um kind of token saving advantages even beyond OCR. s73 00:09:07.120 --> 00:09:13.280 So, why MCP and not just an API or a plugin? s74 00:09:13.280 --> 00:09:26.960 Because we have a dedicated cluster, it was much better to kind of separate the responsibilities and have this as like the backbone to then build more details and more s75 00:09:26.960 --> 00:09:31.760 features further on, for which I will also talk about later. s76 00:09:31.760 --> 00:09:44.920 And everything runs on a server you can control, so through Sai you can pick your uh worker pools, you can pick your uh um GPUs basically, and understand which models are pinned where. s77 00:09:44.920 --> 00:09:57.400 And that flexibility from a cloud perspective was an advantage, and um I wanted to keep it that way, and that's why I went with MCP rather than just a skill. s78 00:09:57.400 --> 00:10:01.480 Um and it exposes the relevant tools. s79 00:10:01.480 --> 00:10:08.080 So, this is roughly how it would work um if it's just the plugin, it's also on your laptop or on device. s80 00:10:08.080 --> 00:10:16.200 In this way, you can kind of offload the same experience, but bring it to the cloud, which is your own cloud, and you can control things there. s81 00:10:16.200 --> 00:10:19.440 So, it's a much cleaner setup. s82 00:10:19.440 --> 00:10:36.640 Um Also, a lot of the examples here are with PII, so personally identifiable information, and there is a tool that gets called for like redacting specific details around documents, which is one of the central use cases of this privacy s83 00:10:36.640 --> 00:10:37.960 aspect. s84 00:10:37.960 --> 00:10:43.320 Um and the MCP setup, it has three tiers. s85 00:10:43.320 --> 00:10:47.560 Um so, there is You can see in the middle the Sci MCP. s86 00:10:47.560 --> 00:11:01.160 There is the MCP client um that's um kind of client and host through co-work or um cloud code or whatever you use, and that communicates with the Sci MCP server. s87 00:11:01.160 --> 00:11:16.880 And then you have actually Sci them cluster that has the um GPUs and the worker pools that do the work by communicating to the um MCP server, and you have different models that are available, s88 00:11:16.880 --> 00:11:26.720 and we also have a good queuing mechanism for this uh GPUs, um and I will go more into that a bit later. s89 00:11:26.720 --> 00:11:38.240 And the idea with MCP, as you know in general, is to have like one interface of communicating, especially if we're building more and more tools with this direction of model routing, s90 00:11:38.240 --> 00:11:43.880 because the demo and the version that we have now is around tool calling. s91 00:11:43.880 --> 00:11:55.320 However, we're building stuff around model routing and understanding which models to pick for different tasks based on um evals and based on costs. s92 00:11:55.320 --> 00:11:58.720 And all of these models would be open source and on your GPU. s93 00:11:58.720 --> 00:12:02.160 So, um there is this important direction as well. s94 00:12:02.160 --> 00:12:05.800 Um so, it's a forward-thinking approach. s95 00:12:05.800 --> 00:12:09.320 Um and there is the host, the client, and the server. s96 00:12:09.320 --> 00:12:19.520 Um you can do this like with any um um setup that you want, like whether it's cloud desktop, cloud code, and so on. s97 00:12:19.560 --> 00:12:24.720 And the communication is kind of done through JSON RPC. s98 00:12:24.720 --> 00:12:39.400 So, there are three like response, request, and a notification just to kind of uh initialize this and have this handshake uh protocol done so that you can authenticate the MCP edge and so on. s99 00:12:39.400 --> 00:12:46.560 And um an important detail is that the server exposes different stuff. s100 00:12:46.560 --> 00:12:52.040 So, in the MCP standard that has changed a bit since its inception. s101 00:12:52.040 --> 00:13:00.200 Um but the main thing is that it exposes tools that then um cloud code would use. s102 00:13:00.200 --> 00:13:15.880 However, in general, there is in kind of the MCP universe, there is a situation where um the client can also expose stuff like sampling or elicitation, which kind of makes this a two-way communication, s103 00:13:15.880 --> 00:13:18.960 so not like a standard client-server setup. s104 00:13:18.960 --> 00:13:25.440 However, in our case, just because this was more of a privacy setup, we do not do this. s105 00:13:25.440 --> 00:13:38.680 So, um basically, the side MCP edge um, only exposes the tools and they can be used, the ones that kind of were initialized in the handshake and that's it. s106 00:13:38.680 --> 00:13:49.000 Um, so it only kind of does this response as it wouldn't um, do this more complex tasks uh, in our case. s107 00:13:49.000 --> 00:13:54.080 And this is the idea that the tools are discovered and you just go from there. s108 00:13:54.080 --> 00:14:04.600 Basically, you uh, kind of have the tools which give you different um, abilities and um, you can run run that. s109 00:14:04.600 --> 00:14:16.320 Another detail is that um, you can do this also locally, so in the MCP universe again, the standard IO is like for local development. s110 00:14:16.320 --> 00:14:22.640 However, we wanted to use the streaming HTTP responses. s111 00:14:22.640 --> 00:14:37.080 This was done in the past by two end points um, that did uh, the streaming and one did the HTTP handshake, but now there is a standard that kind of does this all at once, so that makes things smoother. s112 00:14:37.080 --> 00:14:41.839 And these are the tools that we have, so I'll spend a bit more time on this slide. s113 00:14:41.839 --> 00:14:50.000 So currently, the kind of setup and uh, things you can try are around these models. s114 00:14:50.000 --> 00:14:58.240 So there is this tool that's like docs to markdown and um, that one uses OCR uh, models. s115 00:14:58.240 --> 00:15:01.960 Um, these are the open models that we have currently. s116 00:15:01.960 --> 00:15:08.080 So the idea is it saves token costs, it uses also a very good open source model. s117 00:15:08.080 --> 00:15:23.000 Um, a detail here is that with OCR models, especially the open source models are very good seri- you're not compromising in quality by offloading some of the work to open source um, OCR models. s118 00:15:23.000 --> 00:15:29.080 So, it's not that oh, I'm using open source, I'm sacrificing something. s119 00:15:29.080 --> 00:15:40.440 That's not the case at all, especially here and even in general, open source models are catching up and finding a way to bring them out there, bring them in production is um s120 00:15:40.440 --> 00:15:43.000 quite useful and quite important. s121 00:15:43.000 --> 00:15:50.480 There is extract entities, which would just generate the relevant entities in the docs with different NER models. s122 00:15:50.480 --> 00:15:54.520 There is or different tools around captioning and tagging. s123 00:15:54.520 --> 00:16:03.120 There is also um a capability around uh structure generation, uh summarization, question answering. s124 00:16:03.120 --> 00:16:18.440 And the good thing is also this is open source, so you can also do a PR and add a different tool there right now, and that would be something that kind of we can make make it as a part of this and s125 00:16:18.440 --> 00:16:26.440 um still it will utilize the inference, which is the bigger bottleneck in situations like this. s126 00:16:26.440 --> 00:16:29.400 Um and that would be the idea. s127 00:16:29.400 --> 00:16:33.000 You can kind of clone the repo that I showed before. s128 00:16:33.000 --> 00:16:43.080 And um you can bring up the cluster locally, but also uh if you again go on the website and request that, you'll receive an API key at some point. s129 00:16:43.080 --> 00:16:45.480 And this would be kind of the setup. s130 00:16:45.480 --> 00:16:48.640 You need to get uh the API key. s131 00:16:48.640 --> 00:16:53.760 You can also self-host if you're in for the longer term, let's say. s132 00:16:53.760 --> 00:16:57.160 Uh so, it doesn't have to be our cluster, it can be your cluster. s133 00:16:57.160 --> 00:17:01.880 Um and you just need to set these things up. s134 00:17:01.880 --> 00:17:12.640 I'll also be releasing so, an article that will kind of explain this in a bit more detail, and you can go step by step and run this. s135 00:17:12.640 --> 00:17:14.839 So, no no rush. s136 00:17:14.839 --> 00:17:20.760 Uh you can enjoy the talk and relax and try things out at your own time in many ways. s137 00:17:20.760 --> 00:17:29.360 And um here we have the repo and also the link to kind of getting the uh the compute and the cluster. s138 00:17:29.360 --> 00:17:35.600 So, you can basically request and you get some form that you fill out and then you'll receive the API key. s139 00:17:35.600 --> 00:17:42.840 And this is just the setup that I'll be going through and then trying out. s140 00:17:43.240 --> 00:17:46.000 And hopefully the demo gods are good. s141 00:17:46.000 --> 00:17:49.640 Someone mentioned this as uh a thing at one of the talks. s142 00:17:49.640 --> 00:17:51.760 I like the idea. s143 00:17:51.760 --> 00:17:59.560 Um and yeah, this is kind of just the setup um which I have kind of done to an extent. s144 00:17:59.560 --> 00:18:03.480 And I will now do the demo. s145 00:18:03.480 --> 00:18:06.240 So, I'm going to kind of cloud code. s146 00:18:06.240 --> 00:18:09.880 I am in the relevant packages directory. s147 00:18:09.880 --> 00:18:22.080 So, I can say, "Okay, can you So, I have basically just a set of files that are in an inbox um folder. s148 00:18:22.080 --> 00:18:24.400 So, I'll just show you here. s149 00:18:24.400 --> 00:18:31.400 So, for example, this is the graph and it's like inbox and you have different imagine these are different emails. s150 00:18:31.400 --> 00:18:34.560 So, this is all synthetic data but uh relevant data. s151 00:18:34.560 --> 00:18:50.080 I'm not using the real data that inspired this project but um obviously these are stuff like, "Okay, um there are some emails about an invoice and um some sensitive information s152 00:18:50.080 --> 00:18:53.280 all over and you can just go through this. s153 00:18:53.280 --> 00:19:02.040 There is um some NDA that would have like phone number and social security number, for example. s154 00:19:02.040 --> 00:19:04.840 So, that's how the data looks. s155 00:19:04.840 --> 00:19:19.160 And for example, here you can go and uh maybe say, "Can you uh redact the data from the NDA document. s156 00:19:19.160 --> 00:19:20.960 Something like that. s157 00:19:20.960 --> 00:19:24.960 Uh Cool, we get three credits. s158 00:19:24.960 --> 00:19:29.960 So, um the way things look, one example is this. s159 00:19:29.960 --> 00:19:33.240 So, this is some some of the stuff I tried before. s160 00:19:33.240 --> 00:19:50.280 So, this is the NDA and then you get this kind of like uh redacted uh document that extracts the relevant details, but obviously there is only email and person and the entities are masked. s161 00:19:50.280 --> 00:19:53.200 And all of this is done on your VPC. s162 00:19:53.200 --> 00:19:59.520 So, Cloud Code doesn't see this stuff or whatever agent you're using. s163 00:19:59.520 --> 00:20:05.920 Um and in certain situations you even get better performances with the OCR stuff. s164 00:20:05.920 --> 00:20:14.920 Um and So, for example, you can see here successfully loaded skill. s165 00:20:14.920 --> 00:20:16.720 I found the NDA document. s166 00:20:16.720 --> 00:20:18.920 So, here I've set up the cluster. s167 00:20:18.920 --> 00:20:22.080 So, I just input the endpoint and the API key. s168 00:20:22.080 --> 00:20:28.800 You would need to do this slightly different, but you can totally prompt even Cloud Code uh and so on. s169 00:20:28.800 --> 00:20:40.120 And here um we get Okay, the file contains the first redacted output um redacted 10 spans from this to this. s170 00:20:40.120 --> 00:20:44.520 Um and these are kind of the label counts. s171 00:20:44.520 --> 00:20:46.760 It gives us some info there. s172 00:20:46.760 --> 00:21:02.400 Um and maybe we can do something like Okay, can you pre-process the scans and also redact them to try to see So, this is in natural language. s173 00:21:02.400 --> 00:21:07.560 We try to get the OCR models to work and that's tool to be called. s174 00:21:07.560 --> 00:21:11.080 So, you can also configure the tool calling and all that. s175 00:21:11.080 --> 00:21:22.760 Um And also, there is this uh status line here that counts the tokens saved, but I need to also update this a bit to kind of do it properly, but that's also part of the s176 00:21:22.760 --> 00:21:29.520 um kind of advantage that you can actually see your saving on token costs. s177 00:21:29.520 --> 00:21:34.760 So, I'll pre-process the thing to markdown, then redact it. s178 00:21:34.760 --> 00:21:38.240 Um and it's working. s179 00:21:38.240 --> 00:21:50.320 And for example, we can see I'll just show you some of the files just so that I don't like go through all the stuff all the time. s180 00:21:50.320 --> 00:21:55.560 There is this um let's say contract before. s181 00:21:55.560 --> 00:21:58.000 So, this is how it looks, and this is a scan. s182 00:21:58.000 --> 00:22:04.120 So, I don't know if you can see now cuz it's a clean one, but this is you can see. s183 00:22:04.120 --> 00:22:05.800 So, it's an image. s184 00:22:05.800 --> 00:22:12.080 It's not like text, so it needs um OCR probably. s185 00:22:12.080 --> 00:22:24.040 And then, if we do the after one, this is some You can see markdown file here. s186 00:22:24.040 --> 00:22:29.880 Um that's extracting this stuff, but also redacting them. s187 00:22:29.880 --> 00:22:32.680 So, Oops, cursor. s188 00:22:32.680 --> 00:22:40.200 So, um the idea is that you pre-process the document through the OCR tool calling thing. s189 00:22:40.200 --> 00:22:51.160 You get um the token saving cost, the better context management, and so on, but you also get the redacted file, so personal information is not leaked. s190 00:22:51.160 --> 00:22:57.920 And this is also getting done in the meantime. s191 00:22:58.720 --> 00:23:05.320 So, that's the idea from kind of a functional perspective, but a lot of things are happening in the background. s192 00:23:05.320 --> 00:23:14.600 So, it's not just like um the privacy aspect and this looks easy, but the heavy lifting is done by this part. s193 00:23:14.600 --> 00:23:29.280 So, Superlink um and the inference engine and the cluster is set up in a way that it allows you to have multiple models, have hot swapping uh of models on different GPUs, s194 00:23:29.280 --> 00:23:34.080 um have smarter batching so that you can kind of test out multiple models. s195 00:23:34.080 --> 00:23:48.280 For example, if you have five Laura's uh fine-tuned to different sub areas of your um work, you can kind of hot swap between them and test out which one is the best even. s196 00:23:48.280 --> 00:24:11.360 So, we have a a gateway that um kind of um helps us connect this to different worker pools that each have um this batcher per worker so that the batching is done per worker and that gave us better performance instead of just having all the batching up front. s197 00:24:11.360 --> 00:24:18.240 And this is of course the architecture of the MCP uh solution. s198 00:24:18.240 --> 00:24:31.760 So, how it works with um Sai, so you can see this part on the right um is in your VPC, so both the kind of the MCP work that gets offloaded s199 00:24:31.760 --> 00:24:45.920 to Superlink Sai is in your own private cloud um and this is what gave us the advantage performance-wise, or at least one of the advantages, was the batching technique where s200 00:24:45.920 --> 00:25:02.560 um if you kind of do the batching without having one shared queue that then each of the workers pull from, in our testing and in general performance was lower, so we were able to kind of s201 00:25:02.560 --> 00:25:11.960 do this um setup and have an end-to-end improvement on latency and performance even with different models. s202 00:25:11.960 --> 00:25:27.960 And the idea is that you also want different worker pools because, for example, at some point you might want to try heavier model, but also have another one that can totally run really well on an L4 spot instance, so you mind be like, "Okay, I don't need to waste s203 00:25:27.960 --> 00:25:32.760 um kind of this more expensive hardware." s204 00:25:32.760 --> 00:25:40.320 And also we have auto scaling, so this um can kind of come down when not used and so on. s205 00:25:40.320 --> 00:25:45.817 So, um that's the setup, and there is the hot swapping of models. s206 00:25:45.817 --> 00:25:45.920 [snorts] s207 00:25:45.920 --> 00:26:01.480 So, um especially around smaller models that can fit into one GPU that are doing like named entity recognition or um even smaller generation or embeddings and so on, um s208 00:26:01.480 --> 00:26:08.960 you can benefit a lot from having multiple models on one GPU being switched around and swapped. s209 00:26:09.400 --> 00:26:23.320 And this goes in the direction of kind of what the MCP does and the tool calling aspect where you could build agents and have this inference part offloaded using Superlinked, s210 00:26:23.320 --> 00:26:27.640 and you can get different bundles of tasks set up. s211 00:26:27.640 --> 00:26:41.120 So, basically, document parsing like I explained or embedding content or writing code with um some generation models or extracting images or captions from data and so on. s212 00:26:41.120 --> 00:26:46.400 Uh or even like guardrails and structure generation stuff for policy checking and so on. s213 00:26:46.400 --> 00:26:54.360 So, I'd be also happy for you, so you get more credits if you um kind of build a project and so on. s214 00:26:54.360 --> 00:27:07.760 So, if you contribute a tool um that is maybe around content guarding or policy checking or whatever you find useful, um we'd we'd love to kind of uh have that s215 00:27:07.760 --> 00:27:09.520 as a contribution. s216 00:27:09.520 --> 00:27:21.640 And in some of our tests, we've also uh seen that due to this infrastructure that I explained, um we get very good performance around embedding models. s217 00:27:21.640 --> 00:27:30.720 So, we also have generation models, um but the tests that we have the benchmarks for for now are around embedding models. s218 00:27:30.720 --> 00:27:40.040 Um so, you get kind of um 10x lower price roughly um with very similar quality. s219 00:27:40.040 --> 00:27:53.120 So, you can see there the kind of the best model we had per task for embedding was 99.8% to uh kind of the frontier one. s220 00:27:53.120 --> 00:27:56.800 And this we evaluated this using MTEB tasks. s221 00:27:56.800 --> 00:28:00.080 Um so, yeah, that's um that's the case. s222 00:28:00.080 --> 00:28:04.240 We'll have more benchmarks and details around this soon as well. s223 00:28:04.240 --> 00:28:18.000 But, the point of this is the is that the sometimes, especially in production use cases and in with certain loads, you get the advantage um through the end-to-end improvement. s224 00:28:18.000 --> 00:28:33.120 So, if your infrastructure and architecture is set up well, you basically get the latency improvement and the better performance um even if your per model inference is not optimized. s225 00:28:33.120 --> 00:28:42.680 And we're doing both in our case, but that was a revelation that for production use cases was very helpful to know. s226 00:28:42.880 --> 00:28:52.960 And on the example around LoRAs, we've seen that you could kind of um out Laura's fairly quickly and fine-tune them. s227 00:28:52.960 --> 00:28:57.120 So, we're exploring work on that area. s228 00:28:57.120 --> 00:29:05.640 So, some of the colleagues worked on this German language specific Laura's that were fine-tuned and they were fairly cheap. s229 00:29:05.640 --> 00:29:21.960 So, having a way to do multiple Laura's and test them out in production and deploy them quickly seemed to be quite useful and improved performance in this very Latin niche s230 00:29:21.960 --> 00:29:26.880 use cases like German legal language, let's say. s231 00:29:26.880 --> 00:29:28.520 And this is kind of how s232 00:29:28.520 --> 00:29:29.160 [snorts] s233 00:29:29.160 --> 00:29:32.760 the MCP works. s234 00:29:32.760 --> 00:29:36.200 I mentioned a bit around this. s235 00:29:36.520 --> 00:29:45.560 And there are some failure modes that are already there, but again, this is working and it's well, but there are so many more things to do. s236 00:29:45.560 --> 00:29:56.200 And I'm kind of very happy to have your input and PRs and even just like comments. s237 00:29:56.200 --> 00:30:02.480 So, you can add me on LinkedIn and all that and I'm happy to chat always. s238 00:30:02.480 --> 00:30:12.120 And that's the idea that the same kind of MCP edge handles every document job, so you can do question answering, pulling facts, reading documents. s239 00:30:12.120 --> 00:30:22.480 Um So, yeah, this should also work and there is different again labels extracted and so on. s240 00:30:22.480 --> 00:30:38.800 And you can try this like this I think moved the number, but again, you can test this live and test out different tools with kind of your own demo data. s241 00:30:39.160 --> 00:30:45.360 And I've explained this to a certain point and what it does now. s242 00:30:45.360 --> 00:30:58.960 Um and yeah, there's the token saving aspect and that's very important um and not sending data to external APIs and it all runs in your cloud. s243 00:30:58.960 --> 00:31:03.800 Um and there is a lot of stuff to do kind of in the future. s244 00:31:03.800 --> 00:31:18.400 Um but the operational capability is still there, so we have the Helm charts and the Kubernetes cluster and you can use also Terraform apply to deploy the cluster, which is also open source, so it's it's not s245 00:31:18.400 --> 00:31:25.440 um hidden like kind of uh just the code is open source, but we have basically everything. s246 00:31:25.440 --> 00:31:29.720 Um and this can go beyond just documents. s247 00:31:29.720 --> 00:31:44.560 So, if we look at this idea of having the inference solved as a little box in your whole workflow, it unlocks a lot of opportunities that can go kind of beyond text with other models s248 00:31:44.560 --> 00:31:54.600 looking at transcripts um even graph-based data, generating ontologies and so on by different tool calls, structure generation. s249 00:31:54.600 --> 00:32:14.640 So, there are um different opportunities and um one direction that we're exploring as well is model routing within your own private cloud um which is quite powerful and I think the the field is moving a bit in that direction with s250 00:32:14.640 --> 00:32:26.880 usually teams exploring um this API-based optimization where you have uh a combination of models working together and the end-to-end performance is better. s251 00:32:26.880 --> 00:32:32.200 Um and there is an option to do something similar in this way. s252 00:32:32.200 --> 00:32:42.320 However, this is more like a future roadmap idea, so this is currently not built in, but um the idea is to also pick the models based on the task. s253 00:32:42.320 --> 00:32:51.640 So, the tool calling will not be tied just to the tool itself, but you could also understand which model to pick for the underlying tool. s254 00:32:51.640 --> 00:33:01.600 So, if I've only set up um one OCR model, I could say that, "Okay, for this tool call, I actually want OCR cheap." s255 00:33:01.600 --> 00:33:08.400 or some other sort of tool call that on the task adapts to what I'm trying to do. s256 00:33:08.400 --> 00:33:13.240 And um that's a powerful direction. s257 00:33:13.240 --> 00:33:20.240 And there's also playing around with like intent and there's this mixture of agents approach. s258 00:33:20.280 --> 00:33:35.080 Because if you already Another powerful aspect on why Sci is really good for this is that if you already have the models warm in your GPU, you can hit them much quicker and get the responses s259 00:33:35.080 --> 00:33:35.800 quicker. s260 00:33:35.800 --> 00:33:46.680 And you can do this model routing based on also the the how hot the model is or how kind of what what was evicted last, what wasn't. s261 00:33:46.680 --> 00:33:59.760 So, there is a lot of options also playing around this uh card of latency and trying to figure out what are the best models not just from a general perspective, but from an infrastructure s262 00:33:59.760 --> 00:34:00.800 perspective. s263 00:34:00.800 --> 00:34:15.879 And um doing this is uh quite useful and I I think a lot of um stuff will be done in this in this way um kind of fairly soon. s264 00:34:15.879 --> 00:34:24.583 And um yeah, this is the idea that you can kind of um estimate uh per model to find which models are warm. s265 00:34:24.583 --> 00:34:24.840 [snorts] s266 00:34:24.840 --> 00:34:37.840 Um and then in some cases it would be cheaper to hit a model that's already hot, but more expensive rather than reloading a model that's cheaper, but kind of s267 00:34:37.840 --> 00:34:39.480 not already there in your GPU. s268 00:34:39.480 --> 00:34:47.440 So, this logic can be set up as well to optimize costs and usage beyond the tool level. s269 00:34:47.440 --> 00:35:04.280 Um and I'm just mentioning this stuff just to kind of showcase how powerful um inferences and having good model inference setup um with all the infrastructure around it can actually unlock a lot of um s270 00:35:04.280 --> 00:35:06.120 use cases. s271 00:35:06.120 --> 00:35:16.400 There is also um a potential for doing stuff like um judges talking between each other or like the council of models and so on. s272 00:35:16.400 --> 00:35:30.720 Again, this is more of a future road map, but um this is a direction that also is an option for things like this where, okay, if you want to include some maybe reasoning tool calls or s273 00:35:30.720 --> 00:35:37.840 adjust the response based on like difficulty of tasks, which models are called, for example. s274 00:35:37.840 --> 00:35:49.160 An issue I had when building this was that for one of the summarizing tools, I had a um model that was um that had um thinking tokens. s275 00:35:49.160 --> 00:35:57.000 So, basically, the summarization was either very short or it didn't happen cuz everything was done on the thinking side. s276 00:35:57.000 --> 00:36:04.280 So, you have to also kind of keep in mind what models you use and whether you toggle the thinking tokens or not. s277 00:36:04.280 --> 00:36:18.200 Um But, sometimes reasoning can be beneficial and um it's useful to have a way to do that quickly from an infrastructure side. s278 00:36:18.600 --> 00:36:28.280 And of course, there are some kind of um downsides to having um a router or a council and so on. s279 00:36:28.280 --> 00:36:35.640 However, um um we're also very strong on evals and trying to test models before doing this. s280 00:36:35.640 --> 00:36:40.320 So, um there are different ideas here that can be done. s281 00:36:40.320 --> 00:36:57.640 Um and yeah, um basically we don't want to also spend GPU on some models reasoning forever and then nothing happens um or also have biased models as judges um or kind of optimizing something s282 00:36:57.640 --> 00:37:13.920 that um then becomes kind of by optimizing something you kind of escape from making it as a real target, but it just becomes something that like the models hack and never actually s283 00:37:13.920 --> 00:37:18.120 um use it you can never use that as a proper evaluation. s284 00:37:18.120 --> 00:37:23.280 Um and you can look at our repo. s285 00:37:23.280 --> 00:37:27.760 So, this is the kind of the QR code if you haven't yet. s286 00:37:27.760 --> 00:37:45.600 Um and I think the main takeaway for today is that infrastructure and inference is far more important even for this very API friendly or ergonomic designs where you just work with cloud code and you can kind of offset your s287 00:37:45.600 --> 00:37:57.440 um sensitive data, your specific tasks in different ways and um still the backbone is uh the inference. s288 00:37:57.440 --> 00:38:04.600 And for example, I will go here again. s289 00:38:04.600 --> 00:38:18.360 So, this will be published as well and um you can also find kind of what files are supported, where the Laura aspect feel uh fits, um what are the kind of s290 00:38:18.360 --> 00:38:20.680 the troubleshooting guides and so on. s291 00:38:20.680 --> 00:38:27.680 And also um the most important thing is ask for a kind of your API keys and so on. s292 00:38:27.680 --> 00:38:41.080 We also have a full model catalog where kind of also revamping this to be more detailed and useful, but basically you can have different you can select different output types, different architectures s293 00:38:41.080 --> 00:38:49.000 because in order to optimize the models and the inference you need to adapt models in different bundles. s294 00:38:49.000 --> 00:38:54.000 So you need to kind of all models maybe implement attention differently. s295 00:38:54.000 --> 00:38:58.000 So you cannot run them and run the forward pass in the same way. s296 00:38:58.000 --> 00:39:11.200 So you need to have different model run times for different model families so that this works and there are a lot of details around this and you can see that models are tagged based on like whether they do s297 00:39:11.200 --> 00:39:12.800 encoding or scoring. s298 00:39:12.800 --> 00:39:26.400 So you can also use scoring models that do like re-ranking basically if you even in this use case of the tool calls and the advantage there is that you can s299 00:39:26.400 --> 00:39:40.320 basically play with your own version of retrieval through this MCP and so on and I would really love if you could kind of contribute to this or give us ideas on how to s300 00:39:40.320 --> 00:39:49.240 kind of set this up and build more stuff on top that people find useful. s301 00:39:49.240 --> 00:40:06.480 We did a hackathon in London with Tech Europe and DeepMind recently where a lot of people kind of tested this out and were kind of happy to work both on the retrieval and re-ranking side but also generation. s302 00:40:06.480 --> 00:40:13.920 So having the the way of doing both but being supported by that inference is is very cool. s303 00:40:13.920 --> 00:40:23.320 And again I encourage you to take this um free um kind of cluster infrastructure credits that you can use. s304 00:40:23.320 --> 00:40:31.800 And we also have a lot of examples and um full kind of docs that you can have a look at. s305 00:40:31.800 --> 00:40:45.200 Um basically a quick start and then different model versions and how you can also look at how you can migrate, for example, if you have some workflow done with like OpenAI API, then you're like, "Okay, cool. s306 00:40:45.200 --> 00:40:47.600 I want to test this thing out quickly. s307 00:40:47.600 --> 00:40:48.920 What can I do?" s308 00:40:48.920 --> 00:40:52.760 You have those um options as well. s309 00:40:52.760 --> 00:40:55.320 And uh this is the repo. s310 00:40:55.320 --> 00:41:06.920 So, again, this is the package um and here you have kind of what it exposes um and how the credentials are set up. s311 00:41:06.920 --> 00:41:13.760 So, the the readme here um can be um checked out. s312 00:41:13.760 --> 00:41:15.040 This is the example. s313 00:41:15.040 --> 00:41:22.280 Also, if you contribute with examples and have something more here, that would be very useful. s314 00:41:22.280 --> 00:41:35.480 And um this is where you apply for the credits that um help you to build build things and try things out. s315 00:41:35.480 --> 00:41:43.000 Uh this was very popular at the hackathon, so uh very kind of happy for you to try this out. s316 00:41:43.000 --> 00:41:51.600 And I think since we're slowly approaching the end, I can take some questions as well. s317 00:41:51.600 --> 00:42:00.480 Um you'll just kind of speak and I'll restate and reiterate the question because of the recording. s318 00:42:00.480 --> 00:42:10.960 So, yeah, I'm happy to answer a few things and we can also play around with this more if you have some query that you want to try. s319 00:42:10.960 --> 00:42:12.200 Um yeah. s320 00:42:12.200 --> 00:42:31.600 So, uh using the empty group So, it wouldn't do that basically. s321 00:42:31.600 --> 00:42:35.440 So, the concern is around uh, the question is around privacy. s322 00:42:35.440 --> 00:42:41.080 So, basically you just Claude would just get the redacted version of the document. s323 00:42:41.080 --> 00:42:45.680 So, it wouldn't get the full document that you're inputting there. s324 00:42:45.680 --> 00:42:49.600 So, that's kind of the the setup that we have currently. s325 00:42:49.600 --> 00:42:52.560 So, it would only see that side of things. s326 00:42:52.560 --> 00:42:54.560 Yeah. s327 00:43:09.200 --> 00:43:18.840 Yeah, so I mean, this can be kind of evaluated like whether this happens like whether Claude does this kind of off off-rails situation. s328 00:43:18.840 --> 00:43:30.880 Definitely, that can be kind of tested more and more, but in the test that I've done like if you host the kind of MCP and everything else is on the virtual private cloud, s329 00:43:30.880 --> 00:43:34.200 this seemed to have worked without kind of the data leaking. s330 00:43:34.200 --> 00:43:43.200 But, I'm sure there can be more guardrails to kind of reinforce this at that initial stage or in the worst case even you could say, "Okay, I'm concerned. s331 00:43:43.200 --> 00:43:48.000 I can build my own harness and use an open-source exactly." s332 00:43:48.000 --> 00:43:49.040 Yeah, yeah. s333 00:43:49.040 --> 00:43:49.600 Yeah, yeah. s334 00:43:49.600 --> 00:44:01.320 And in that way even um, like you could still save on tokens and get better context even if like this is Claude code in my example, but it can be any harness. s335 00:44:01.320 --> 00:44:09.400 If you implement the tool calling well, I think this can be done, but happy happy to get a contribution around this as well. s336 00:44:09.400 --> 00:44:13.680 Um, Could be very useful to have this like pre-processing layer as well. s337 00:44:13.680 --> 00:44:17.920 Um Yeah, another Yeah. s338 00:44:39.880 --> 00:44:41.880 Mhm. s339 00:44:42.720 --> 00:44:43.160 Yeah. s340 00:44:43.160 --> 00:44:44.640 Yeah. s341 00:44:44.640 --> 00:44:55.680 So, again the question is kind of like how do we set this up to make sure that the privacy is well preserved, right? s342 00:44:56.680 --> 00:45:10.800 So, on on the on the side side of things, so the cluster in this case is like what we have, but that can be your own cluster. s343 00:45:10.800 --> 00:45:17.480 So, through the repo you can set up your own cluster in your own Yes, exactly. s344 00:45:17.480 --> 00:45:20.280 Yeah, so you can configure the cluster in any way you want. s345 00:45:20.280 --> 00:45:29.400 I'm just using a managed cluster that we have for the purpose of the demo and the example and we use that for example for hackathons and so on. s346 00:45:29.400 --> 00:45:34.120 So, we host the hardware, but you can totally do this yourself. s347 00:45:34.120 --> 00:45:46.600 So, it's self-hosted and in the as I said in the repo you have the ability to do Terraform apply and have all the infrastructure set up and if you have the GPUs and so on, you basically s348 00:45:46.600 --> 00:45:52.600 um yeah, have that set up and it's totally your own. s349 00:45:56.160 --> 00:45:58.520 Sorry, again? s350 00:46:00.080 --> 00:46:03.600 Does the MCP have a file? s351 00:46:04.640 --> 00:46:12.600 So it just sends it to the main worker and there the file is processed. s352 00:46:12.600 --> 00:46:16.440 So kind of there is that yeah. s353 00:46:23.760 --> 00:46:33.280 And yeah, you can again I think at the entrance there is the free compute that you can get to try out. s354 00:46:33.280 --> 00:46:37.200 Again, if you want to self-host, you can still self-host. s355 00:46:37.200 --> 00:46:41.120 It's not um strict in that sense. s356 00:46:41.120 --> 00:46:44.920 We just do this so that things can go quicker. s357 00:46:44.920 --> 00:46:51.120 But again, you don't have to do it in a way that you use our cluster. s358 00:46:51.120 --> 00:46:53.560 You can self-host. s359 00:46:57.160 --> 00:47:00.920 And yeah, we're also you can kind of add me on LinkedIn. s360 00:47:00.920 --> 00:47:09.320 Again, we can chat there and um Thank you for attending.