{
  "video": {
    "id": "Akm1sqvWG4A",
    "title": "Bypassing the Multimodal Tax: Hybrid RAG, SQL RRF & UI Telemetry - Abed Matini, Ogilvy",
    "duration": 2748,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": "Online"
  },
  "analysis": {
    "video_id": "Akm1sqvWG4A",
    "title": "Bypassing the Multimodal Tax: Hybrid RAG, SQL RRF & UI Telemetry - Abed Matini, Ogilvy",
    "one_liner": "A framework-free, fully local RAG stack — Docling → markdown → Postgres hybrid search (vector + BM25) → a 0.5B Qwen model — that avoids paying the 'multimodal tax' of dropping raw PDFs into a cloud chatbot, demoed live as an HR FAQ assistant.",
    "summary": "Matini argues that drag-and-dropping documents into an LLM chat burns tokens before a single question is asked and hides how the document was chunked, so you can't tell why an answer is wrong. His alternative is to structure first on local CPU: Docling converts PDFs, Word, PowerPoint and screenshots to markdown, a chosen chunking strategy writes clean chunks into PostgreSQL, and retrieval mixes cosine-distance vector search with BM25 keyword search before a top-k slice goes to the model. He demos a running HR FAQ assistant with an admin page for chunking strategies, per-answer chunk references, code-level guardrails that block medical and prompt-injection queries before the LLM, and Langfuse traces showing model, chunks returned and latency — all on Ollama with no GPU, no paid framework and runnable in one command on GitHub Codespaces.",
    "key_points": [
      "Two stated problems: uploading a document into a cloud chatbot spends tokens before any question is asked, and a production chatbot accumulates too many tools (vector DB, keyword search, semantic search) to manage.",
      "Stack is deliberately plain: Python + FastAPI, React front end, PostgreSQL as the vector store, Docker, Ollama for local chat and embedding models, Langfuse for telemetry — no paid framework, no GPU, CPU is good enough, one command on GitHub Codespaces.",
      "Docling runs locally to convert raw PDF/Word/PowerPoint/image into markdown, which is then chunked and embedded into Postgres — so you can see exactly what the LLM sees.",
      "Live comparison: dumping the whole 28-page employee handbook produces junk chunks ('acknowledgement', 'signature date') that slow the system, reduce accuracy and increase hallucination; the same content chunked by heading into question/answer pairs returns the right answer (90% of individual employee premium, 75% of dependent premium) with a clickable chunk reference you can debug.",
      "Four chunking strategies shown in the admin UI: heading-based, paragraph, fixed 512 characters with 64 characters of overlap (32 either side), and sentence-based; a screenshot of a maintenance email is converted image → text → .md and sentence-grouped, then answered with the exact maintenance window and the screenshot as reference.",
      "'Agents' are plain Python functions, not LLM calls — current date, product info, deeper search — chosen for speed (multi-agent loops would make users wait 20–30 seconds locally), full control, no hallucination, and coverage by a test suite; agent mode is a toggle and does cost latency and referenceability.",
      "Hybrid retrieval is mandatory because semantic nearest-neighbour returns 'close' matches while product SKUs, brand names and medication names need exact keyword hits via BM25; the two rankings are fused and top-k is tuned by use case — more for product catalogues (or 50 products of a brand never get shown), fewer for medical where liability is higher. The demo caps at top 2.",
      "Guardrails and prompt-injection defences live in code before the LLM is ever called — intent regexes, term dictionaries and an LLM classifier — which is why the system prompt is only a few sentences; blocked queries return a canned medical-escalation message and the block list is covered by rigid tests.",
      "Model finding: he started on Qwen2.5 7B, found it slow and too wordy, and dropped to Qwen2.5 0.5B Instruct (~400MB) plus a small BGE embedding model — two models total. Vetting the data first makes the smallest model sufficient, less hallucinatory, and more willing to say it doesn't have the information."
    ],
    "takeaways": [
      "Stop uploading raw documents into the chat. Convert to markdown with Docling on local CPU, chunk deliberately, and store it — you stop paying per-upload tokens and you gain visibility into what was actually indexed.",
      "Pick the chunking strategy per document: headings (or hand-authored Q&A) for structured FAQs, paragraphs for prose, fixed 512/64-overlap only for unstructured data you can't clean, sentence groups for screenshots and short emails.",
      "Run both vector and BM25 and fuse the rankings rather than choosing one, and tune top-k to the domain — high for product catalogues so inventory is discoverable, low for medical or compliance answers.",
      "Move deterministic work — dates, calculations, lookups, injection and topic blocking — out of prompts and into Python functions you can unit test, and stop risky queries before they reach the LLM rather than instructing the LLM to refuse.",
      "Try the smallest model first: if retrieval is clean, a 0.5B instruct model answers well, runs without a GPU, and hallucinates less than a larger one."
    ],
    "topics": [
      "rag",
      "chunking",
      "hybrid-search",
      "local-llms",
      "observability",
      "guardrails",
      "prompt-injection",
      "postgres"
    ],
    "tools": [
      "Docling",
      "PostgreSQL",
      "FastAPI",
      "Python",
      "React",
      "Docker",
      "Ollama",
      "Langfuse",
      "Qwen2.5 0.5B Instruct",
      "Qwen2.5 7B",
      "BGE embedding model",
      "BM25",
      "GitHub Codespaces",
      "Ogilvy"
    ],
    "quotes": [
      {
        "text": "as soon as we upload these documents, we going to basically lose some of our tokens that we supposed to have only for processing these documents without even asking a question",
        "at": "01:39",
        "url": "https://www.youtube.com/watch?v=Akm1sqvWG4A&t=99s"
      },
      {
        "text": "this going to slow down and reduce the accuracy of our chat. It's going to increase hallucination.",
        "at": "12:49",
        "url": "https://www.youtube.com/watch?v=Akm1sqvWG4A&t=769s"
      },
      {
        "text": "The rest if something can be done by a Python function, let's a Python function then run it for you. Then you have also the full control over the function. Won't be any hallucination",
        "at": "26:58",
        "url": "https://www.youtube.com/watch?v=Akm1sqvWG4A&t=1618s"
      },
      {
        "text": "Interesting finding from my side was you don't need the biggest model. You if you can vet your data before sending it to your LLM, a smaller smallest model can sort you out",
        "at": "42:48",
        "url": "https://www.youtube.com/watch?v=Akm1sqvWG4A&t=2568s"
      }
    ],
    "words": 7390
  },
  "summary_url": "/#Akm1sqvWG4A",
  "transcript": {
    "html": "/transcripts/Akm1sqvWG4A.html",
    "txt": "/transcripts/Akm1sqvWG4A.txt",
    "vtt": "/transcripts/Akm1sqvWG4A.vtt"
  }
}