{ "video": { "id": "Ndp5VaqFrl0", "title": "Turning My Obsidian Vault Into a Local AI Engineer — Filip Makraduli, Superlinked", "duration": 2846, "upload_date": null, "channel": "AI Engineer", "source": "AI Engineer", "event_format": "" }, "analysis": { "video_id": "Ndp5VaqFrl0", "title": "Turning My Obsidian Vault Into a Local AI Engineer — Filip Makraduli, Superlinked", "one_liner": "Filip Makraduli runs an MCP server from Claude Code that sends sensitive, token-heavy document work (OCR, PII redaction, entity extraction) to open-source models on a GPU cluster in your own VPC, so Claude only sees the compact, redacted markdown that comes back.", "summary": "The project started from an Obsidian vault of markdown files and three goals: privacy, lower token costs, and adaptability (trying several open-source models and task-specific LoRAs). Makraduli built an MCP integration for Superlinked's open-source inference engine, which he calls 'Sci'. Claude Code calls tools such as docs-to-markdown (OCR) and PII redaction, and those tools run on worker pools in your own VPC. Claude only receives the redacted, compact output. His main argument is that the hard part is the inference infrastructure, not data storage or the agent interface: hot-swapping models, batching per worker, and routing based on which models are already warm decide cost and latency.", "key_points": [ "The MCP setup has three tiers: an MCP client/host (Claude Code, Cowork, Claude Desktop), the Sci MCP server, and the Sci cluster with GPU worker pools and a queuing mechanism. It communicates over JSON-RPC and uses streamable HTTP instead of stdio. For privacy, the server only exposes tools. It does not use client-side features like sampling or elicitation.", "Current tools: docs-to-markdown (open-source OCR models), extract entities (NER models), captioning/tagging, structured generation, summarization, and question answering. On OCR, his claim is that 'you're not compromising in quality' by moving the work to open-source models.", "Token-saving example: pre-processing something like a 10-page PDF of scans and images with OCR returns compact markdown to the main agent. That means fewer tokens and less context clutter. A status line in Claude Code counts tokens saved.", "Demo on synthetic inbox data (invoices, an NDA containing phone and social security numbers): Claude Code redacted the NDA ('redacted 10 spans', with label counts), then OCR'd scanned contracts into markdown and redacted them. All of this ran in the VPC, so Claude never saw the raw data.", "Infrastructure: a gateway routes to worker pools, and each worker has its own batcher. In their tests this beat a single shared up-front queue, improving end-to-end latency. Worker pools let light models run on cheap L4 spot instances while heavier models get bigger GPUs. The cluster autoscales down when idle and hot-swaps small models (NER, embeddings, small generation models) on one GPU.", "Embedding benchmarks on MTEB tasks: about 10x lower price, with the best model per task reaching 99.8% of the frontier model. His conclusion is that good architecture can deliver the end-to-end gains even when per-model inference isn't optimized.", "Colleagues fine-tuned cheap German-language LoRAs that improved niche use cases such as German legal language. Hot-swapping several LoRAs makes it possible to test them in production.", "On the roadmap, not built yet: per-task model routing inside the private cloud (e.g. choosing 'OCR cheap'), routing based on which models are warm, and councils or judges of models. He warns about judge bias, reward hacking, and models that spend GPU reasoning forever." ], "takeaways": [ "Offload sensitive or token-heavy pre-processing (OCR, redaction, NER) from your coding agent to self-hosted open-source models through an MCP server, so the frontier model only gets compact, redacted context.", "Expose tools only from a privacy-focused MCP server (no sampling or elicitation), and run the server and its models inside your own VPC. Deploy them with the open-source Terraform, Helm, and Kubernetes setup.", "Batch per worker instead of using one shared queue, and split hardware into worker pools so cheap models run on cheap GPUs (e.g. L4 spot instances).", "Check whether a model emits thinking tokens before using it for a task like summarization. In his case the summary came back very short or empty because all the output went into thinking.", "Cost-aware routing: calling a pricier model that is already loaded on the GPU can be cheaper than reloading a cheaper one." ], "topics": [ "mcp", "privacy", "ocr", "pii-redaction", "inference", "open-source-models", "token-costs", "model-routing" ], "tools": [ "Obsidian", "Superlinked", "Superlinked Sci (inference engine)", "Claude Code", "Claude Cowork", "Claude Desktop", "Fable", "Model Context Protocol (MCP)", "JSON-RPC", "Docker", "Kubernetes", "Helm", "Terraform", "MTEB", "NVIDIA L4", "LoRA", "OpenAI API", "Tech Europe", "DeepMind", "LinkedIn" ], "quotes": [ { "text": "So, it's not that oh, I'm using open source, I'm sacrificing something. That's not the case at all", "at": "15:24", "url": "https://www.youtube.com/watch?v=Ndp5VaqFrl0&t=924s" }, { "text": "in some cases it would be cheaper to hit a model that's already hot, but more expensive rather than reloading a model that's cheaper", "at": "34:24", "url": "https://www.youtube.com/watch?v=Ndp5VaqFrl0&t=2064s" }, { "text": "the summarization was either very short or it didn't happen cuz everything was done on the thinking side", "at": "35:50", "url": "https://www.youtube.com/watch?v=Ndp5VaqFrl0&t=2150s" }, { "text": "I think the main takeaway for today is that infrastructure and inference is far more important", "at": "37:28", "url": "https://www.youtube.com/watch?v=Ndp5VaqFrl0&t=2248s" } ], "words": 7406 }, "summary_url": "/#Ndp5VaqFrl0", "transcript": { "html": "/transcripts/Ndp5VaqFrl0.html", "txt": "/transcripts/Ndp5VaqFrl0.txt", "vtt": "/transcripts/Ndp5VaqFrl0.vtt" } }