{ "video": { "id": "wzWNYDY7toc", "title": "From Raw Documents to AI-Ready Data — Leo Platzer & Jeff Koss", "duration": 1297, "upload_date": null, "channel": "AI Engineer", "source": "AI Engineer", "event_format": "" }, "analysis": { "video_id": "wzWNYDY7toc", "title": "From Raw Documents to AI-Ready Data — Leo Platzer & Jeff Koss", "one_liner": "Before RAG, deduplicate your documents and drop stale ones: in DZ Labs' evaluation, a corpus where 30% of files were stale or duplicates could fill up to 80% of the agent's context with useless information, and cleaning it roughly doubled recall and lifted task completion by 10–15%.", "summary": "Jeff and Leo Platzer from DZ Labs (acquired by Calibbra last summer) argue that unstructured-document RAG pilots stall when they grow from about 40 hand-picked files to more than 80,000. At that size teams can't find the relevant files, don't know where sensitive data is, and can't trust document quality: duplicates, conflicting versions and freshness. Using a fictional manufacturer called North River and a SharePoint legal-ops chatbot, they demo AI-driven taxonomy tagging, sensitivity scanning, quality filtering and auto-refreshing 'data slices'. Leo argues that deduplication and freshness improve retrieval no matter the use case, and shows the DZ SDK generating per-folder context MD files so coding agents can navigate a SharePoint site without opening every file.", "key_points": [ "The scenario is North River Manufacturing: 40,000 employees in 20 countries, with millions of documents in SharePoint, building a legal-ops chatbot to answer questions on termination clauses, which agreements expire this or next quarter, and exclusivity terms.", "A proof of concept on about 40 pre-curated files (parsing, chunking, OCR, embedding, vector DB plus knowledge graph) worked well but broke at 80,000+ files. The problems were finding the files, locating sensitive data, and trusting quality.", "For documents, Jeff says data quality is not the classic six dimensions such as completeness, conformity and accuracy. It means duplicate information, conflicting information and freshness. Another customer, a telecom with 6–7 SMEs, couldn't keep up with curating 20,000 HTML files by hand.", "Demo: a taxonomy (a 'tagging tree') is built from user-defined tags, a reusable tag library, or tags the AI suggests from a use-case description. Each tag has an LLM prompt, optional classification values, CSV upload, and conditional sub-tags (e.g. contract type only for legal/contract files). Results show evidence, confidence scores, and thumbs up/down for few-shot learning. Two filters took 339 files down to 20.", "The sensitivity scan covers PII and PHI, AI-based and pattern-based with context. Asked to detect Finnish national IDs, it wrote the regex and added Finnish context words to cut false positives.", "Quality features: grouping duplicates and conflicting documents with an AI suggestion of which to keep, freshness filters (last X days or a date range), and required-metadata rules, e.g. 339 files becoming perhaps 200 into a 'data slice'. Slices refresh on a schedule so the SDK always feeds the vector DB fresh files and metadata.", "Claimed result for North River: data prep cut from four months to days, higher chatbot accuracy from injecting metadata through the SDK, and lower compliance risk because sensitive data was filtered out.", "Leo's evaluation, on a multihop RAG benchmark rather than a coding task: if 30% of a 1,000-file corpus is stale or duplicate, up to 80% of the context the agent uses can be stale. Cleaning it 'pretty much' doubled recall@1 and improved task completion by 10–15%. Live demo: the DZ SDK fetches a SharePoint site's metadata and writes a context MD per folder plus an index (document types, key topics, when to use it, example questions), so Claude Code or Codex can skip opening files and spend fewer tokens." ], "takeaways": [ "Before scaling a RAG pilot, deduplicate and remove stale documents. Leo argues this helps whatever the use case, while relevance filtering depends on knowing the use case.", "Treat document quality as duplicates, conflicting versions and freshness rather than structured-data dimensions, and decide explicitly which version of a document is the truth.", "Tag documents with LLM-generated metadata from a structured taxonomy (with conditional sub-tags), check the evidence and confidence scores, and use the metadata to filter and enrich retrieval.", "Scan for sensitive data (PII/PHI and region-specific IDs) before indexing, so it can't leak through the chatbot.", "For coding agents, put per-folder context summaries and an index in front of the raw files, so agents read a context MD instead of every document and use fewer tokens." ], "topics": [ "rag", "unstructured data", "data quality", "metadata tagging", "data governance", "context engineering", "pii detection", "agents" ], "tools": [ "DZ Labs", "Calibbra", "DZ SDK", "SharePoint", "Informatica", "Databricks", "Gartner", "Slack", "Claude Code", "Codex" ], "quotes": [ { "text": "your top K will always be full of pretty much irrelevant information", "at": "15:42", "url": "https://www.youtube.com/watch?v=wzWNYDY7toc&t=942s" }, { "text": "it can happen that up to 80% of your actual context your agent uses to answer a question is filled up with stale information.", "at": "17:03", "url": "https://www.youtube.com/watch?v=wzWNYDY7toc&t=1023s" }, { "text": "duplication and freshness is true no matter what.", "at": "18:15", "url": "https://www.youtube.com/watch?v=wzWNYDY7toc&t=1095s" }, { "text": "they were able to go from four months of data prep work to down a days.", "at": "14:16", "url": "https://www.youtube.com/watch?v=wzWNYDY7toc&t=856s" } ], "words": 4003 }, "summary_url": "/#wzWNYDY7toc", "transcript": { "html": "/transcripts/wzWNYDY7toc.html", "txt": "/transcripts/wzWNYDY7toc.txt", "vtt": "/transcripts/wzWNYDY7toc.vtt" } }