{ "video": { "id": "0I07YAuF8xA", "title": "From Ingestion to Agents: How AI Teams Build on Document Intelligence — Adit Abraham, Reducto", "duration": 1349, "upload_date": null, "channel": "AI Engineer", "source": "AI Engineer", "event_format": "In person" }, "analysis": { "video_id": "0I07YAuF8xA", "title": "From Ingestion to Agents: How AI Teams Build on Document Intelligence — Adit Abraham, Reducto", "one_liner": "Clean inputs matter even more for agents than for RAG. Reducto's advice is to split document parsing into cheap sub-million-parameter CV layout models plus VLMs, check the output with 'agentic OCR' token-level edits, format the data for whatever consumes it, and run evals at every stage of the pipeline.", "summary": "Reducto has processed many billions of documents for AI-native companies and large enterprises. The talk argues that as AI moves from RAG-style information synthesis to agents doing multi-step work, bad document inputs compound through the pipeline, and PDFs remain surprisingly hard to parse. Abraham covers when to use traditional computer vision versus VLMs, agentic OCR for verification, formatting choices for LLMs versus embedding models, classifying and splitting documents, and agent harnesses that solve problems like line-chart extraction and very large form extraction. He closes by arguing that evals should underpin every decision, at every step of the pipeline.", "key_points": [ "Serge's 'GDP PDF' benchmark has Fable, the current frontier model, at about 30%. When models get Reducto's structured parse alongside the original PDF, results improve across Gemini, Anthropic and OpenAI models. GPT 5.5 and Opus then beat Fable out of the box, and they use fewer reasoning tokens, which lowers latency.", "Traditional CV is underrated. Object detection has improved thanks to autonomous-vehicle research, and sub-million-parameter models running on CPU detect document layout well, cheaply and more deterministically. VLMs add semantics and handle handwriting and the long tail in ways traditional OCR never could.", "'Agentic OCR' works like speculative decoding or Cursor's fast edits. It applies token-level corrections (period vs. comma, zero vs. O) on top of a CV + VLM parse instead of asking Gemini to re-transcribe. Free-form next-token generation introduces new errors: seeing the word 'total', a model may add up the table itself instead of copying the value as written.", "Table formatting is chosen per table: markdown for simple tables and HTML for complex ones with merged cells, since HTML costs a lot of tokens. For retrieval, embedding models struggle to match a natural-language query to a blob of HTML tags and numbers. So they generate a natural-language description of the table for embedding and pass the HTML to the LLM for reasoning.", "Classifying and splitting documents is underappreciated. Stuffing in too much context lowers quality, not just token economics. Example: paper-mail packets hundreds of pages long with interleaved content.", "Agent harnesses solve problems that used to be unsolved. For line charts (common at hedge funds), no model can extract the data in a single shot. An agent with a code interpreter that plots its own reconstruction and fixes mistakes iteratively produces the full data table.", "For structured extraction of forms with tens of thousands of fields (e.g. CBP forms), a parent agent sets validation criteria for sub-agents to follow. A new micro1 benchmark shows frontier models with max reasoning are precise but silently drop rows (poor recall), while dedicated document tools have better recall and lower precision. Only an agent harness got both.", "Customers are moving away from deterministic pipelines toward giving agents a file system and a CLI. The agent decides what to read, and output is split into a content field plus metadata such as bounding boxes for citations. Reducto is moving next toward document editing and generation." ], "takeaways": [ "Decompose the parsing problem and use the right tool for each part (CV for layout, VLMs for semantics) to reach the best trade-off between accuracy, cost and latency.", "Add an agentic verification and correction layer that makes constrained token-level edits, rather than prompting an LLM to rewrite OCR output.", "Format data for its consumer: markdown or HTML chosen per table for LLMs, and natural-language descriptions of tables for embedding and retrieval.", "Use classify and split steps to route documents and pass only the relevant snippets, instead of dumping in the whole context.", "Eval every stage separately (parsing, retrieval, final formatting, end-to-end agent performance) and monitor production data, which will differ from your curated eval sets." ], "topics": [ "document-parsing", "ocr", "agents", "rag", "evals", "vlms", "data-extraction", "retrieval" ], "tools": [ "Reducto", "Harvey", "Rogo", "Serge (GDP PDF benchmark)", "micro1", "Fable", "GPT 5.5", "Opus", "Gemini", "OpenAI", "Anthropic", "Cursor", "Claude Code", "Google Drive", "Box" ], "quotes": [ { "text": "PDFs are surprisingly still a very hard problem", "at": "05:33", "url": "https://www.youtube.com/watch?v=0I07YAuF8xA&t=333s" }, { "text": "They'll see the word total and if the human made a mistake in that table models will actually sometimes go through and add up the values in the table themselves.", "at": "10:06", "url": "https://www.youtube.com/watch?v=0I07YAuF8xA&t=606s" }, { "text": "at the end of the day eval should underpin all of your decisions", "at": "19:06", "url": "https://www.youtube.com/watch?v=0I07YAuF8xA&t=1146s" }, { "text": "agentic verification is the biggest step change that the industry has had for a while", "at": "21:18", "url": "https://www.youtube.com/watch?v=0I07YAuF8xA&t=1278s" } ], "words": 5112 }, "summary_url": "/#0I07YAuF8xA", "transcript": { "html": "/transcripts/0I07YAuF8xA.html", "txt": "/transcripts/0I07YAuF8xA.txt", "vtt": "/transcripts/0I07YAuF8xA.vtt" } }