Turning My Obsidian Vault Into a Local AI Engineer — Filip Makraduli, Superlinked

AI Engineer · 47 min · 360 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:15Hello, hello.
00:16Um, I guess we're starting.
00:18I know some of you are still joining in, but I'll go on and you can jump in at any time.
00:26Uh, so you might have seen the title of this talk.
00:30Um, and what I have on the slides here is slightly different, but I promise it's the same direction and the same general idea.
00:39So, it all started for me wanting to build a use case that's around privacy and token costs, basically.
00:49So, um, I started with Obsidian.
00:52I had some markdown files there.
00:54Um, and as you know, it's like you have this nice, um, graphs and it's all good.
01:02Looks very nice.
01:04However, um, I had certain issues beyond just organizing the files.
01:12So, a lot of people do this knowledge graphs and so on and like memory for agents and that's quite good.
01:20However, I needed to approach this differently because there was a concern around privacy.
01:27So, when using agents, when using cloud code, when using co-work, uh, you can expose your files and for some institutions, for some people, this was, um, a requirement that,
01:42um, needed to be met.
01:44So, the goal was to keep things private and self-hosted, but also make them quite usable and friendly so that people could actually do these things.
01:56And, um, there was kind of this openness to fine-tune, to be very adaptable, to use open-source models, and, um I found that the solution was, um in Superlinked, which is
02:13a model inference, um engine that's open-source.
02:17So, it has both a cluster and a model inference solution.
02:21And, I will be talking about that today, and how I managed to kind of merge these concepts together, and, um create something that leans on these three things.
02:35So, the first main idea was I wanted to make Claude Code a bit cheaper and save on tokens.
02:43So, we know how with the release of Fable, there initially there was this, um API-only pricing, and that kind of, uh maybe disappointed a few people, or the token economics got a bit more attention now.
02:59So, that was one thing.
03:00Especially for document processing, sensitive document processing, where you need to do OCR, there are a lot of scans, images, and stuff like this, the token cost can ramp up.
03:11So, I wanted to find a way to kind of do this, but also having smaller artifacts means that you get better context, so you kind of limit context loss in a way, and you don't like
03:26fill out, um the context with data that can be processed in some way else.
03:34Um, and I wanted to make this private.
03:37So, yes, for certain situations, it's fine to paste everything into Claude Code or have that API working there.
03:45But, what if you could kind of self-host and have your own VPC or your own cloud to preprocess your sensitive documents and then return something back.
03:54So, this was another target of mine when building this.
03:59And also I wanted adaptability, so it wasn't just um privacy and lowering token costs, but also having the chance to test out multiple models, maybe host Laura's that are very task specific
04:14because with this very sensitive use cases, usually there is some sort of fine-tuning or adaptation towards that niche use case.
04:24So, I had to do that as well.
04:28And the setup um involves Superlinked um and Sci the repo, so you can for the purpose of today, you can go to um the repo which is fully open source, so you can find it on GitHub.
04:47It's Superlinked Sci.
04:49Uh you can Yeah.
04:51So, this is it.
04:52This is the logo.
04:53And I'll be talking about um an integration that's uh Sorry, not here.
05:01So, an integra an MCP integration which is basically a package that we did, which is this one here.
05:07So, you can also have a look at this subdirectory.
05:12So, this is where most of the talk will be about, but I'll also mention why actually the inference was the bottleneck when building something like this.
05:21So, it was easy to figure out how to store the data, maybe in Obsidian and so on and have these connections between the data.
05:30Um That was the easy part.
05:34I could also use an API.
05:35However, if I wanted to do open source, if I wanted to do open source inference at scale and testing out multiple models, being able to deploy this in production,
05:47things got a bit difficult and that's why the infrastructure advantage that Sci offers, the fact that you're able to kind of deploy both a cluster and the models themselves
06:01was a very big advantage in my case.
06:05Um here you can see this um Obsidian example.
06:10So I'll work with kind of some of these things.
06:13Uh but you can also try this yourself.
06:16Um however, I will be using a managed cluster.
06:21We cannot um I cannot give that to you right now live.
06:26However, you can go to our website, which is this one, and here you have this apply for in inference grant.
06:34So you can go there and you'll receive your um API key that you can try and you can do things quicker.
06:41If not, you can do things locally cuz there is a Docker container and all that.
06:45Uh it will just take a bit slower to do these things, but I'd prefer if you kind of listen to me now and then try things after, but also in parallel if you want.
06:55And we also have this very cool um kind of QR codes memorabilia stuff where you can get an even better cluster, as I think, and you can find them here
07:07uh after the talk.
07:10So, um as I said, the idea was to have this privacy element and have a way to offload.
07:19So basically, I didn't want to disrupt my normal usage day-to-day.
07:24Um so I'm just working in cloud code and then if there's a specific task that needs to be done on my virtual private clouds that needs to be processed, that's sensitive, that costs a lot of tokens,
07:37I can just offload it and continue my work.
07:40So, the natural direction was to build some sort of um skill or plugin.
07:47But, this was even elevated more by building a whole MCP server that's that gets called in cloud code and there's that element of like using skills, but also having
08:01actually a dedicated edge that can actually do the heavy lifting.
08:07And you can also benefit from Superlinks inference, which I will talk about later on kind of how we've done this and what are the advantages of using um Superlinks inference engine.
08:22So, it saves tokens because for example, a number would be to have like maybe 10 pages of a PDF that has images, different details in there.
08:33So, if you pre-process that with good open-source OCR models and then return to your main agent harness or cloud code or whatever you're using, you get the context, but also
08:47benefit on the lower token spent and better context and so on.
08:52So, that's the general idea, especially for OCR this is apparent.
08:58However, there are multiple um kind of token saving advantages even beyond OCR.
09:07So, why MCP and not just an API or a plugin?
09:13Because we have a dedicated cluster, it was much better to kind of separate the responsibilities and have this as like the backbone to then build more details and more
09:26features further on, for which I will also talk about later.
09:31And everything runs on a server you can control, so through Sai you can pick your uh worker pools, you can pick your uh um GPUs basically, and understand which models are pinned where.
09:44And that flexibility from a cloud perspective was an advantage, and um I wanted to keep it that way, and that's why I went with MCP rather than just a skill.
09:57Um and it exposes the relevant tools.
10:01So, this is roughly how it would work um if it's just the plugin, it's also on your laptop or on device.
10:08In this way, you can kind of offload the same experience, but bring it to the cloud, which is your own cloud, and you can control things there.
10:16So, it's a much cleaner setup.
10:19Um Also, a lot of the examples here are with PII, so personally identifiable information, and there is a tool that gets called for like redacting specific details around documents, which is one of the central use cases of this privacy
10:36aspect.
10:37Um and the MCP setup, it has three tiers.
10:43Um so, there is You can see in the middle the Sci MCP.
10:47There is the MCP client um that's um kind of client and host through co-work or um cloud code or whatever you use, and that communicates with the Sci MCP server.
11:01And then you have actually Sci them cluster that has the um GPUs and the worker pools that do the work by communicating to the um MCP server, and you have different models that are available,
11:16and we also have a good queuing mechanism for this uh GPUs, um and I will go more into that a bit later.
11:26And the idea with MCP, as you know in general, is to have like one interface of communicating, especially if we're building more and more tools with this direction of model routing,
11:38because the demo and the version that we have now is around tool calling.
11:43However, we're building stuff around model routing and understanding which models to pick for different tasks based on um evals and based on costs.
11:55And all of these models would be open source and on your GPU.
11:58So, um there is this important direction as well.
12:02Um so, it's a forward-thinking approach.
12:05Um and there is the host, the client, and the server.
12:09Um you can do this like with any um um setup that you want, like whether it's cloud desktop, cloud code, and so on.
12:19And the communication is kind of done through JSON RPC.
12:24So, there are three like response, request, and a notification just to kind of uh initialize this and have this handshake uh protocol done so that you can authenticate the MCP edge and so on.
12:39And um an important detail is that the server exposes different stuff.
12:46So, in the MCP standard that has changed a bit since its inception.
12:52Um but the main thing is that it exposes tools that then um cloud code would use.
13:00However, in general, there is in kind of the MCP universe, there is a situation where um the client can also expose stuff like sampling or elicitation, which kind of makes this a two-way communication,
13:15so not like a standard client-server setup.
13:18However, in our case, just because this was more of a privacy setup, we do not do this.
13:25So, um basically, the side MCP edge um, only exposes the tools and they can be used, the ones that kind of were initialized in the handshake and that's it.
13:38Um, so it only kind of does this response as it wouldn't um, do this more complex tasks uh, in our case.
13:49And this is the idea that the tools are discovered and you just go from there.
13:54Basically, you uh, kind of have the tools which give you different um, abilities and um, you can run run that.
14:04Another detail is that um, you can do this also locally, so in the MCP universe again, the standard IO is like for local development.
14:16However, we wanted to use the streaming HTTP responses.
14:22This was done in the past by two end points um, that did uh, the streaming and one did the HTTP handshake, but now there is a standard that kind of does this all at once, so that makes things smoother.
14:37And these are the tools that we have, so I'll spend a bit more time on this slide.
14:41So currently, the kind of setup and uh, things you can try are around these models.
14:50So there is this tool that's like docs to markdown and um, that one uses OCR uh, models.
14:58Um, these are the open models that we have currently.
15:01So the idea is it saves token costs, it uses also a very good open source model.
15:08Um, a detail here is that with OCR models, especially the open source models are very good seri- you're not compromising in quality by offloading some of the work to open source um, OCR models.
15:23So, it's not that oh, I'm using open source, I'm sacrificing something.
15:29That's not the case at all, especially here and even in general, open source models are catching up and finding a way to bring them out there, bring them in production is um
15:40quite useful and quite important.
15:43There is extract entities, which would just generate the relevant entities in the docs with different NER models.
15:50There is or different tools around captioning and tagging.
15:54There is also um a capability around uh structure generation, uh summarization, question answering.
16:03And the good thing is also this is open source, so you can also do a PR and add a different tool there right now, and that would be something that kind of we can make make it as a part of this and
16:18um still it will utilize the inference, which is the bigger bottleneck in situations like this.
16:26Um and that would be the idea.
16:29You can kind of clone the repo that I showed before.
16:33And um you can bring up the cluster locally, but also uh if you again go on the website and request that, you'll receive an API key at some point.
16:43And this would be kind of the setup.
16:45You need to get uh the API key.
16:48You can also self-host if you're in for the longer term, let's say.
16:53Uh so, it doesn't have to be our cluster, it can be your cluster.
16:57Um and you just need to set these things up.
17:01I'll also be releasing so, an article that will kind of explain this in a bit more detail, and you can go step by step and run this.
17:12So, no no rush.
17:14Uh you can enjoy the talk and relax and try things out at your own time in many ways.
17:20And um here we have the repo and also the link to kind of getting the uh the compute and the cluster.
17:29So, you can basically request and you get some form that you fill out and then you'll receive the API key.
17:35And this is just the setup that I'll be going through and then trying out.
17:43And hopefully the demo gods are good.
17:46Someone mentioned this as uh a thing at one of the talks.
17:49I like the idea.
17:51Um and yeah, this is kind of just the setup um which I have kind of done to an extent.
17:59And I will now do the demo.
18:03So, I'm going to kind of cloud code.
18:06I am in the relevant packages directory.
18:09So, I can say, "Okay, can you So, I have basically just a set of files that are in an inbox um folder.
18:22So, I'll just show you here.
18:24So, for example, this is the graph and it's like inbox and you have different imagine these are different emails.
18:31So, this is all synthetic data but uh relevant data.
18:34I'm not using the real data that inspired this project but um obviously these are stuff like, "Okay, um there are some emails about an invoice and um some sensitive information
18:50all over and you can just go through this.
18:53There is um some NDA that would have like phone number and social security number, for example.
19:02So, that's how the data looks.
19:04And for example, here you can go and uh maybe say, "Can you uh redact the data from the NDA document.
19:19Something like that.
19:20Uh Cool, we get three credits.
19:24So, um the way things look, one example is this.
19:29So, this is some some of the stuff I tried before.
19:33So, this is the NDA and then you get this kind of like uh redacted uh document that extracts the relevant details, but obviously there is only email and person and the entities are masked.
19:50And all of this is done on your VPC.
19:53So, Cloud Code doesn't see this stuff or whatever agent you're using.
19:59Um and in certain situations you even get better performances with the OCR stuff.
20:05Um and So, for example, you can see here successfully loaded skill.
20:14I found the NDA document.
20:16So, here I've set up the cluster.
20:18So, I just input the endpoint and the API key.
20:22You would need to do this slightly different, but you can totally prompt even Cloud Code uh and so on.
20:28And here um we get Okay, the file contains the first redacted output um redacted 10 spans from this to this.
20:40Um and these are kind of the label counts.
20:44It gives us some info there.
20:46Um and maybe we can do something like Okay, can you pre-process the scans and also redact them to try to see So, this is in natural language.
21:02We try to get the OCR models to work and that's tool to be called.
21:07So, you can also configure the tool calling and all that.
21:11Um And also, there is this uh status line here that counts the tokens saved, but I need to also update this a bit to kind of do it properly, but that's also part of the
21:22um kind of advantage that you can actually see your saving on token costs.
21:29So, I'll pre-process the thing to markdown, then redact it.
21:34Um and it's working.
21:38And for example, we can see I'll just show you some of the files just so that I don't like go through all the stuff all the time.
21:50There is this um let's say contract before.
21:55So, this is how it looks, and this is a scan.
21:58So, I don't know if you can see now cuz it's a clean one, but this is you can see.
22:04So, it's an image.
22:05It's not like text, so it needs um OCR probably.
22:12And then, if we do the after one, this is some You can see markdown file here.
22:24Um that's extracting this stuff, but also redacting them.
22:29So, Oops, cursor.
22:32So, um the idea is that you pre-process the document through the OCR tool calling thing.
22:40You get um the token saving cost, the better context management, and so on, but you also get the redacted file, so personal information is not leaked.
22:51And this is also getting done in the meantime.
22:58So, that's the idea from kind of a functional perspective, but a lot of things are happening in the background.
23:05So, it's not just like um the privacy aspect and this looks easy, but the heavy lifting is done by this part.
23:14So, Superlink um and the inference engine and the cluster is set up in a way that it allows you to have multiple models, have hot swapping uh of models on different GPUs,
23:29um have smarter batching so that you can kind of test out multiple models.
23:34For example, if you have five Laura's uh fine-tuned to different sub areas of your um work, you can kind of hot swap between them and test out which one is the best even.
23:48So, we have a a gateway that um kind of um helps us connect this to different worker pools that each have um this batcher per worker so that the batching is done per worker and that gave us better performance instead of just having all the batching up front.
24:11And this is of course the architecture of the MCP uh solution.
24:18So, how it works with um Sai, so you can see this part on the right um is in your VPC, so both the kind of the MCP work that gets offloaded
24:31to Superlink Sai is in your own private cloud um and this is what gave us the advantage performance-wise, or at least one of the advantages, was the batching technique where
24:45um if you kind of do the batching without having one shared queue that then each of the workers pull from, in our testing and in general performance was lower, so we were able to kind of
25:02do this um setup and have an end-to-end improvement on latency and performance even with different models.
25:11And the idea is that you also want different worker pools because, for example, at some point you might want to try heavier model, but also have another one that can totally run really well on an L4 spot instance, so you mind be like, "Okay, I don't need to waste
25:27um kind of this more expensive hardware."
25:32And also we have auto scaling, so this um can kind of come down when not used and so on.
25:40So, um that's the setup, and there is the hot swapping of models.
25:45[snorts]
25:45So, um especially around smaller models that can fit into one GPU that are doing like named entity recognition or um even smaller generation or embeddings and so on, um
26:01you can benefit a lot from having multiple models on one GPU being switched around and swapped.
26:09And this goes in the direction of kind of what the MCP does and the tool calling aspect where you could build agents and have this inference part offloaded using Superlinked,
26:23and you can get different bundles of tasks set up.
26:27So, basically, document parsing like I explained or embedding content or writing code with um some generation models or extracting images or captions from data and so on.
26:41Uh or even like guardrails and structure generation stuff for policy checking and so on.
26:46So, I'd be also happy for you, so you get more credits if you um kind of build a project and so on.
26:54So, if you contribute a tool um that is maybe around content guarding or policy checking or whatever you find useful, um we'd we'd love to kind of uh have that
27:07as a contribution.
27:09And in some of our tests, we've also uh seen that due to this infrastructure that I explained, um we get very good performance around embedding models.
27:21So, we also have generation models, um but the tests that we have the benchmarks for for now are around embedding models.
27:30Um so, you get kind of um 10x lower price roughly um with very similar quality.
27:40So, you can see there the kind of the best model we had per task for embedding was 99.8% to uh kind of the frontier one.
27:53And this we evaluated this using MTEB tasks.
27:56Um so, yeah, that's um that's the case.
28:00We'll have more benchmarks and details around this soon as well.
28:04But, the point of this is the is that the sometimes, especially in production use cases and in with certain loads, you get the advantage um through the end-to-end improvement.
28:18So, if your infrastructure and architecture is set up well, you basically get the latency improvement and the better performance um even if your per model inference is not optimized.
28:33And we're doing both in our case, but that was a revelation that for production use cases was very helpful to know.
28:42And on the example around LoRAs, we've seen that you could kind of um out Laura's fairly quickly and fine-tune them.
28:52So, we're exploring work on that area.
28:57So, some of the colleagues worked on this German language specific Laura's that were fine-tuned and they were fairly cheap.
29:05So, having a way to do multiple Laura's and test them out in production and deploy them quickly seemed to be quite useful and improved performance in this very Latin niche
29:21use cases like German legal language, let's say.
29:26And this is kind of how
29:28[snorts]
29:29the MCP works.
29:32I mentioned a bit around this.
29:36And there are some failure modes that are already there, but again, this is working and it's well, but there are so many more things to do.
29:45And I'm kind of very happy to have your input and PRs and even just like comments.
29:56So, you can add me on LinkedIn and all that and I'm happy to chat always.
30:02And that's the idea that the same kind of MCP edge handles every document job, so you can do question answering, pulling facts, reading documents.
30:12Um So, yeah, this should also work and there is different again labels extracted and so on.
30:22And you can try this like this I think moved the number, but again, you can test this live and test out different tools with kind of your own demo data.
30:39And I've explained this to a certain point and what it does now.
30:45Um and yeah, there's the token saving aspect and that's very important um and not sending data to external APIs and it all runs in your cloud.
30:58Um and there is a lot of stuff to do kind of in the future.
31:03Um but the operational capability is still there, so we have the Helm charts and the Kubernetes cluster and you can use also Terraform apply to deploy the cluster, which is also open source, so it's it's not
31:18um hidden like kind of uh just the code is open source, but we have basically everything.
31:25Um and this can go beyond just documents.
31:29So, if we look at this idea of having the inference solved as a little box in your whole workflow, it unlocks a lot of opportunities that can go kind of beyond text with other models
31:44looking at transcripts um even graph-based data, generating ontologies and so on by different tool calls, structure generation.
31:54So, there are um different opportunities and um one direction that we're exploring as well is model routing within your own private cloud um which is quite powerful and I think the the field is moving a bit in that direction with
32:14usually teams exploring um this API-based optimization where you have uh a combination of models working together and the end-to-end performance is better.
32:26Um and there is an option to do something similar in this way.
32:32However, this is more like a future roadmap idea, so this is currently not built in, but um the idea is to also pick the models based on the task.
32:42So, the tool calling will not be tied just to the tool itself, but you could also understand which model to pick for the underlying tool.
32:51So, if I've only set up um one OCR model, I could say that, "Okay, for this tool call, I actually want OCR cheap."
33:01or some other sort of tool call that on the task adapts to what I'm trying to do.
33:08And um that's a powerful direction.
33:13And there's also playing around with like intent and there's this mixture of agents approach.
33:20Because if you already Another powerful aspect on why Sci is really good for this is that if you already have the models warm in your GPU, you can hit them much quicker and get the responses
33:35quicker.
33:35And you can do this model routing based on also the the how hot the model is or how kind of what what was evicted last, what wasn't.
33:46So, there is a lot of options also playing around this uh card of latency and trying to figure out what are the best models not just from a general perspective, but from an infrastructure
33:59perspective.
34:00And um doing this is uh quite useful and I I think a lot of um stuff will be done in this in this way um kind of fairly soon.
34:15And um yeah, this is the idea that you can kind of um estimate uh per model to find which models are warm.
34:24[snorts]
34:24Um and then in some cases it would be cheaper to hit a model that's already hot, but more expensive rather than reloading a model that's cheaper, but kind of
34:37not already there in your GPU.
34:39So, this logic can be set up as well to optimize costs and usage beyond the tool level.
34:47Um and I'm just mentioning this stuff just to kind of showcase how powerful um inferences and having good model inference setup um with all the infrastructure around it can actually unlock a lot of um
35:04use cases.
35:06There is also um a potential for doing stuff like um judges talking between each other or like the council of models and so on.
35:16Again, this is more of a future road map, but um this is a direction that also is an option for things like this where, okay, if you want to include some maybe reasoning tool calls or
35:30adjust the response based on like difficulty of tasks, which models are called, for example.
35:37An issue I had when building this was that for one of the summarizing tools, I had a um model that was um that had um thinking tokens.
35:49So, basically, the summarization was either very short or it didn't happen cuz everything was done on the thinking side.
35:57So, you have to also kind of keep in mind what models you use and whether you toggle the thinking tokens or not.
36:04Um But, sometimes reasoning can be beneficial and um it's useful to have a way to do that quickly from an infrastructure side.
36:18And of course, there are some kind of um downsides to having um a router or a council and so on.
36:28However, um um we're also very strong on evals and trying to test models before doing this.
36:35So, um there are different ideas here that can be done.
36:40Um and yeah, um basically we don't want to also spend GPU on some models reasoning forever and then nothing happens um or also have biased models as judges um or kind of optimizing something
36:57that um then becomes kind of by optimizing something you kind of escape from making it as a real target, but it just becomes something that like the models hack and never actually
37:13um use it you can never use that as a proper evaluation.
37:18Um and you can look at our repo.
37:23So, this is the kind of the QR code if you haven't yet.
37:27Um and I think the main takeaway for today is that infrastructure and inference is far more important even for this very API friendly or ergonomic designs where you just work with cloud code and you can kind of offset your
37:45um sensitive data, your specific tasks in different ways and um still the backbone is uh the inference.
37:57And for example, I will go here again.
38:04So, this will be published as well and um you can also find kind of what files are supported, where the Laura aspect feel uh fits, um what are the kind of
38:18the troubleshooting guides and so on.
38:20And also um the most important thing is ask for a kind of your API keys and so on.
38:27We also have a full model catalog where kind of also revamping this to be more detailed and useful, but basically you can have different you can select different output types, different architectures
38:41because in order to optimize the models and the inference you need to adapt models in different bundles.
38:49So you need to kind of all models maybe implement attention differently.
38:54So you cannot run them and run the forward pass in the same way.
38:58So you need to have different model run times for different model families so that this works and there are a lot of details around this and you can see that models are tagged based on like whether they do
39:11encoding or scoring.
39:12So you can also use scoring models that do like re-ranking basically if you even in this use case of the tool calls and the advantage there is that you can
39:26basically play with your own version of retrieval through this MCP and so on and I would really love if you could kind of contribute to this or give us ideas on how to
39:40kind of set this up and build more stuff on top that people find useful.
39:49We did a hackathon in London with Tech Europe and DeepMind recently where a lot of people kind of tested this out and were kind of happy to work both on the retrieval and re-ranking side but also generation.
40:06So having the the way of doing both but being supported by that inference is is very cool.
40:13And again I encourage you to take this um free um kind of cluster infrastructure credits that you can use.
40:23And we also have a lot of examples and um full kind of docs that you can have a look at.
40:31Um basically a quick start and then different model versions and how you can also look at how you can migrate, for example, if you have some workflow done with like OpenAI API, then you're like, "Okay, cool.
40:45I want to test this thing out quickly.
40:47What can I do?"
40:48You have those um options as well.
40:52And uh this is the repo.
40:55So, again, this is the package um and here you have kind of what it exposes um and how the credentials are set up.
41:06So, the the readme here um can be um checked out.
41:13This is the example.
41:15Also, if you contribute with examples and have something more here, that would be very useful.
41:22And um this is where you apply for the credits that um help you to build build things and try things out.
41:35Uh this was very popular at the hackathon, so uh very kind of happy for you to try this out.
41:43And I think since we're slowly approaching the end, I can take some questions as well.
41:51Um you'll just kind of speak and I'll restate and reiterate the question because of the recording.
42:00So, yeah, I'm happy to answer a few things and we can also play around with this more if you have some query that you want to try.
42:10Um yeah.
42:12So, uh using the empty group So, it wouldn't do that basically.
42:31So, the concern is around uh, the question is around privacy.
42:35So, basically you just Claude would just get the redacted version of the document.
42:41So, it wouldn't get the full document that you're inputting there.
42:45So, that's kind of the the setup that we have currently.
42:49So, it would only see that side of things.
42:52Yeah.
43:09Yeah, so I mean, this can be kind of evaluated like whether this happens like whether Claude does this kind of off off-rails situation.
43:18Definitely, that can be kind of tested more and more, but in the test that I've done like if you host the kind of MCP and everything else is on the virtual private cloud,
43:30this seemed to have worked without kind of the data leaking.
43:34But, I'm sure there can be more guardrails to kind of reinforce this at that initial stage or in the worst case even you could say, "Okay, I'm concerned.
43:43I can build my own harness and use an open-source exactly."
43:48Yeah, yeah.
43:49Yeah, yeah.
43:49And in that way even um, like you could still save on tokens and get better context even if like this is Claude code in my example, but it can be any harness.
44:01If you implement the tool calling well, I think this can be done, but happy happy to get a contribution around this as well.
44:09Um, Could be very useful to have this like pre-processing layer as well.
44:13Um Yeah, another Yeah.
44:39Mhm.
44:42Yeah.
44:43Yeah.
44:44So, again the question is kind of like how do we set this up to make sure that the privacy is well preserved, right?
44:56So, on on the on the side side of things, so the cluster in this case is like what we have, but that can be your own cluster.
45:10So, through the repo you can set up your own cluster in your own Yes, exactly.
45:17Yeah, so you can configure the cluster in any way you want.
45:20I'm just using a managed cluster that we have for the purpose of the demo and the example and we use that for example for hackathons and so on.
45:29So, we host the hardware, but you can totally do this yourself.
45:34So, it's self-hosted and in the as I said in the repo you have the ability to do Terraform apply and have all the infrastructure set up and if you have the GPUs and so on, you basically
45:46um yeah, have that set up and it's totally your own.
45:56Sorry, again?
46:00Does the MCP have a file?
46:04So it just sends it to the main worker and there the file is processed.
46:12So kind of there is that yeah.
46:23And yeah, you can again I think at the entrance there is the free compute that you can get to try out.
46:33Again, if you want to self-host, you can still self-host.
46:37It's not um strict in that sense.
46:41We just do this so that things can go quicker.
46:44But again, you don't have to do it in a way that you use our cluster.
46:51You can self-host.
46:57And yeah, we're also you can kind of add me on LinkedIn.
47:00Again, we can chat there and um Thank you for attending.