Structuring the Unstructured - Cedric Clyburn, Red Hat

AI Engineer · 20 min · 189 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:00Hey, hey, welcome.
00:00My name is Cedric Clyburn.
00:02I'm an open source engineer here at Red Hat, and I think we can all agree that context is the most important aspect to building an AI application or an agent, right?
00:11It's the reason that harnesses have become so popular in order to manage the LLM's context.
00:15But, the thing is, no matter what model or agent that you're using, there is so much data that we're not able to use properly because it's in unstructured formats.
00:25I'm talking everything from PDFs to presentations to contracts and technical docs, even meeting notes, scanned documents, diagrams, tables, images, and more.
00:35And uh sorry, I know that's a lot, but you understand what I mean, right?
00:38All this data needs to be transformed into something that an LLM can actually understand.
00:43And that's why by the end of this session, you'll understand how to extract structure between raw enterprise documents and use it to power better downstream AI systems like rag and agent.
00:53So, let's get started.
00:55Now, I think Jensen from Nvidia made this point super clear at his keynote that unstructured data is becoming this new context layer for AI.
01:02And the reality though, for many teams, and I know this personally working at Red Hat, that PDFs and data are spread across dozens of different systems.
01:12So, we've got a lot to cover today.
01:14As you might know, a large majority of the world's data is unstructured, and so no matter what model you're using, if you're working with data like PDFs and unstructured types of formats,
01:23this is a bit tricky to work with.
01:25Because there are solutions out there, but they might be proprietary or require you to send your private data to someone else's server.
01:33And for not just text, how do we take documents and their graphs or tables and images to formats that LLMs can understand like markdown or JSON?
01:42And I'm going to show you how in the session today because we're going to be using an open source tool, part of the Linux Foundation, that is called Docling,
01:49and learn about extraction, parsing, chunking, and much more.
01:53And I've got some live demos for you, so we're going to have some fun.
01:56And just in case you'd like it, we have the session slides here on the right and a little overview of the specifics I'll be showing you in today's session.
02:04But, without further ado, let's get started.
02:06So, why is there a need for advanced document processing?
02:10As I briefly mentioned before, you might have a lot of technical documentation or meeting minutes or different types of documents and invoices that you need to use and maybe rag or different type of applications
02:22where the uh context is provided to an LLM.
02:26So, whether it's rag or retrieval augmented generation to answer questions based on this data, or you're using this to fine-tune a new specialized model, well, data is this key ingredient behind those applications.
02:38And it doesn't matter if you're using, you know, NVIDIA acceleration or an open-source or proprietary model, that data and the way you process it is the key determining factor
02:50in whether your answer is going to be correct or incorrect for the user or customer at the end of the day.
02:55And that's what's most important.
02:58And how important is it?
02:59Well, I have this viral tweet from earlier where 20 scientific papers now feature a new nonsensical term that doesn't exist because AI misinterpreted a very old article that was scanned and taken to a PDF,
03:13merging two different words from two different columns in this PDF.
03:17And because researchers are using these models in order to help them write, now we have different types of scientific papers that all feature this word and are even being cited by other people.
03:28And so, that's how important it is to make sure that the data that we're processing is processed in a way that's accurate and not hallucinated and able to be used confidently
03:38in our applications that we're delivering to users and customers.
03:41So, it's quite important.
03:43Now, if we were to use a tool like Docling that I can run on my own machine, you could see that these two words are quite far away from each other and shouldn't have been combined in the first place.
03:54But, that's how we're going to learn about extracting this text here in a second.
03:58Now, if we were to try a simple PDF parser for a PDF like this that includes um a table here, it has an image, there's captions, and there's regular sections of text,
04:08well, we might get an answer like this here on the right in markdown.
04:11You know, this might be very fast and cheap to run even on CPU, but the issue is is that a lot of this text has been truncated, has been merged,
04:22and isn't decipherable even by me as a human.
04:25And if I sent this to a model, I don't think I could trust that the model could extract specifics from, say for example, this table.
04:31Because the table has been kind of just spit out linearly, and this information isn't fit for most use cases where I need to ask questions or have an agent do validation
04:43and extraction on this source data.
04:46So, this isn't going to cut it, right?
04:48There's undesired page headers, we don't understand the table, and where's the content from the image, right?
04:52It's not even there.
04:54When we're using frontier models, this is um kind of not bad, right?
04:59But, it's quite expensive.
05:00If I'm sending this to a model that's maybe $30 per million output tokens, you can see how this can get quite expensive as I scale this up to dozens or hundreds or, in a lot of cases, thousands of PDFs that organizations
05:13have to work through to use an AI application.
05:16And the difference is between maybe a 5.1 of a model that was depreciated in the 5.2 version of a model make it tricky to have structured output that's consistent every single time.
05:27And so, while it might be good quality and I can see that most of this looks accurate in the exported markdown, we might be susceptible to hallucinations because models are non-deterministic,
05:37and this is really tricky at scale.
05:40And so, what is the middle ground?
05:42Well, that's where Docling comes in.
05:44It's a fast and cheap, and most importantly, local CLI and library that I can use to take various types of input sources and convert this to markdown, JSON, and a Pydantic
05:56data type that I can use in my applications and that I can scale up if I have thousands of different types of formats that need to be used or translated to something like markdown.
06:08And so we rendered this as HTML, but you can see here we've got this export of the specific table that we had here previously.
06:16Because as a PDF, this data type is spread out and it's proprietary.
06:21So, it's hard to extract this type of data from the source PDF here.
06:26And I'll show you how Docling does it by using a combination of OCR and specific vision models that extract the format and allow me to do things like structure output if I only, say for example, want a specific column to be outputted
06:41from a content source.
06:42So, it's really cool.
06:43And I've been using it at Red Hat.
06:45My team uses it because we have thousands of PDFs that we need to work with, specifically from product documentation.
06:52But also, we have content like images that we want to extract using vision models.
06:56And what Docling does with a pip install Docling is allow me to convert single documents or websites or anything else to markdown or whatever file type I want and be able to work with the page layout so I don't lose the consistency
07:10in the structure of the source document.
07:13And there's a lot of other integrations and for situations where maybe you need to run this locally or you don't want to pay for a service or you have an air-gapped environment.
07:23So, you're able to do this using this open-source project that is part of the Linux Foundation.
07:28So, before I show you the quick demo that I want to highlight the project, I want to talk a little bit about scale and cost.
07:36As I just mentioned it there earlier, but there is a public use case I want to show you from Leandro at Hugging Face where he compared a source of common crawl PDFs
07:46and where he did a little bit of pre-work on them and actually extracted the structure using OCR and using DocLing in order to remove certain parts and clean this up for the fine PDFs
07:57export which is thousands of tokens from PDFs around the web that you could use for training a model etc. But the two comparisons he did using GPU and using CPU for DocLing
08:09allowed him to do this at 50 times of a cost savings compared to VLMs and OCR naively.
08:19And so this is what's really cool is that you can really scale this up and he did this on CPU and not needing GPU which is really cool.
08:27And it's not just document conversion, right?
08:29This example was getting things ready from PDFs, but let's say we have images, right?
08:34This image with vision language model was able to be described and annotated, right?
08:40From this specific image right here, now we have all of this really important context that can be used in a rag application as an example, but also just for the end user to be able to understand what's happening in this image.
08:53And say for example, the SME doesn't work there anymore at the organization, now we have a way to understand what this is with the help of an LLM whether proprietary
09:03or local.
09:04And then finally, it's not just document conversion and annotation, but also structured output.
09:10Say for example, I have this invoice and I need to extract specifically the bill number, the total invoice price, the name of the sender, we're able to get that in a format
09:19that is structured.
09:21So I don't have to worry about, "Hey, it's going to pull in like the headings and titles."
09:25No, I just want the, you know, total invoice price and the bill number and I can get that in a format that's pydantic, but also just super simple if I'm just trying to get a a things out of a huge document.
09:36So let's hop over to the demo.
09:38Let me show you what I'm talking about.
09:39And by the way, we're going to be using this Docling workshop repository for the demos today.
09:44So, feel free to check it out.
09:45In this first example, we're going to be using Docling to convert a popular file type like a PDF because remember that data is the foundation for all AI systems.
09:53And in order to leverage that data, we have to properly ingest different file formats with accuracy.
09:58And without doing that, we can lose information or information can become unreadable from tables and diagrams and images.
10:05So, with Docling, what we're able to do with a simple pip install down here is start processing some of these different file types.
10:12So, I'll show you how we do that.
10:13First off, we're going to be importing essential components like the document converter as well as some other dependencies.
10:20And I'll show you the simplest way, which is to just start out with a PDF that we have online, which is Docling's own research paper.
10:28And here you can see we have different titles and subtitles.
10:31We have components such as images and captions that we have here.
10:34And at the bottom of this PDF, we also have something like a table here that we need to extract as well as images that might be helpful for our AI application or agent.
10:44So, let me come back to the notebook here and I'll show you how we do this simple conversion by using the Docling's document converter and exporting this to markdown.
10:53So, here you can see is a rough example of how fast this can be for a eight-page PDF to be able to export this in a way that an LLM can start using, right?
11:03But at the same time, this is a Pydantic data type.
11:06So, you can see we can explore this PDF and the number of pages and tables and see what is on which page of this PDF and export this into a variety of different formats
11:16like markdown, HTML, dictionary, and much more.
11:20But the real value here is not just with basic text and columns, but working with tables.
11:25So, this PDF here has a variety of different tables that we want to be able to extract.
11:30And so, by doing a converter for this specific document, we can then extract these tables here and be able to export this to a data frame so we can render this out in our Jupiter notebook here.
11:42So as I run this cell, you can see that we've exported eight different tables from that source PDF and we can list these but also be able to get these in a format ready to use in a rag application
11:53or just to query with our LLM.
11:55So we've taken a look at how to pull text and tables from a PDF, but what about visualizing and extracting images from that source document as well?
12:04Now, for this specific example, what we'll do is set up a PDF pipeline that will allow us to scale up the images and push this into a document converter.
12:13So now when I go to inspect the images and picture content in the cells, we can see this nicely mapped out where we have the picture, so the source image,
12:22the caption of that, and all of the embedded text elements that we could use in some type of retrieval augmented generation application to ask questions about, "Hey, what's happening in these different photos
12:33in our source PDF?"
12:35I think what would help here is also to be able to visualize the document layout by using the bounding boxes provided by the layout visualizer here.
12:43So here what we're going to do is visualize all of the different elements and components that can be extracted from, say for example, that source PDF.
12:52So section headers or text or subtitles or different components such as that photo that we just pulled and extracted from the PDF.
13:00So this is one of the models that Doc Lee provides which can be used for situations where you might have personal identifiable information from a customer that you want to remove from a source document type before you extract that into your application.
13:13We can also use vision language models in order to enrich the source images and diagrams that might be in these document types using something like O llama or a third-party LLM.
13:23So what we're going to do here is set up a PDF pipeline that's going to use a local running Granite model and say, "Hey, give us a detailed description of what's happening in this image."
13:34And with that document converter, we're going to go ahead and display that enriched document by calling the OpenAI endpoint with Ollama that we have running locally at the completions endpoint.
13:45And so here we have an annotated caption of what's happening with that DocQuery pipeline image that we were taking a look at earlier.
13:51Where originally we were just pulling the caption, but now we can use a vision language model to describe what's happening in these photos.
13:57And with all this additional information, this can help us to build a really solid RAG pipeline to where we can do questioning and answering over our source data.
14:07And for this example, I want to show you what's known as chunkless RAG or agentic RAG using DocQuery.
14:13And now by starting off with our document outline, so processing a PDF with DocQuery like we just did, we can allow an LLM to be able to pick the most relevant
14:23part of the document that is related to the user's question and pull that full text from the DocQuery document itself to try to answer that question for the user.
14:33And this can run in a agentic loop.
14:35And what's really important here is that we're doing RAG but without having to use a chunker or embedding model or vector database, etc., etc. So the index ends up being the markdown outline of the document.
14:48Now when the user asks a question, the entire retrieval index typically would be thousands of vectors in a database where we would do semantic similarity to make sure, "Hey, these sections are similar to the user's question."
14:59But for us, what we're going to do is be able to see a markdown outline of the document with each section summary outline.
15:06So if the LLM is looking for something about getting started with DocQuery, well it can just pull from this reference of text to say that DocQuery can be installed from PyPI.
15:15And that's the entire retrieval index.
15:17So let's say we have a query, "What are the main AI models used in DocQuery?"
15:22Well we're setting up a rag agent here to be able to iterate on that specific question about five times.
15:29So, we see that there are 20 sections available when the user is asked the question, and we're going to search for that specific part of text that talks about the AI models
15:39and be able to determine, "Hey, is this relevant to answering the question?"
15:43And you can see here the final answer in one iteration was pulled from that source material without having to go in the vector database but instead search that docling document structure
15:53for the specific text.
15:55So, it's a quite interesting way to be able to answer users' questions through this chunklist retrieval augmented generation pattern.
16:02And while this first example might have been simple, what we can do is also pull the IBM 2025 annual report into the context here, which has 418 sections, so it's much larger, and ask the question like, "Hey, what was Red Hat's revenue growth in 2025,
16:19and how did to an overall software segment?"
16:22And so, here we're going to be iterating multiple times to figure out, "Hey, is this section relevant to the user's question?"
16:29And if not, we need to pull in more information.
16:32And so, that is how chunkless rag can work in a situation where you're using a tool like docling.
16:37But what happens when we have hundreds or hundreds of thousands of PDFs that we want to have processed?
16:43Well, this is where we can deploy docling as a REST API service using something that's known as docling serve.
16:49This allows us to scale things up and to run this as a microservice as a container or through Kubernetes.
16:55So, when we set things up here, we're going to do a pip install docling serve, and we're going to be able to serve this from a CLI with docling serve on a specific port,
17:05or once we have that server started, be able to send things to that specific endpoint with different types of options and arguments like, "Hey, do we want OCR?
17:15Do we want a specific back end?
17:16Or do we want images annotated?
17:18And so, that's how we can scale things up and allow this API endpoint to be able to handle hundreds or thousands of different document types at a time.
17:27And let's say, for example, that you're trying to build an AI agent, you can also use the Docling MCP server.
17:32So, this allows us to automate things with our AI agent and give it the capabilities that Docling has through that model context protocol and allow us to standardize the communication between, say, for example, Claude Coder Continue
17:47in our developer CLI to the MCP server, which can handle the document processing for us without us having to know all of those different arguments and commands.
17:55So, it makes it quite easy.
17:57And the example I show here is by using one of the Claude models on my own Mac itself and connecting this to my VS Code instance.
18:05So, let's understand the tools that are available.
18:07For the MCP server, we have conversion tools, generation tools, say, for example, if I want to process a specific part of a PDF, and manipulation tools.
18:17And this is all provided to the LLM and the agent that we're going to be using with the MCP server.
18:23So, here I'm just checking that my local MLX server to run an LLM that's running, and it looks like we've got Quinn 3.6 here.
18:31And we're going to verify that the Docling MCP server is also running, which would be done using UVX here.
18:37And so, we'll do that in this cell here to make sure that the MCP server is available.
18:41Now, in Claude Coder or Codex or another type of AI application, we're going to install the extension.
18:49So, for us, this means adding an MCP server in the config.yaml.
18:54And now, at this point, we can use a model and an MCP server to do things such as, "Hey, convert this document and give me a summary."
19:03Or, create a document with a section of action items and pull in a list from another PDF and export that as markdown.
19:10So, we can use all of those Docling components through the MCP server in order to agentically process and parse these documents using an AI agent like Cursor or Claude Code or one of the many open-source
19:24options that are out there.
19:26Now, let's head back to the slides and wrap things up.
19:28So, let's put it all together.
19:29With Docling, we've seen that we can take a PDF into format like Markdown or JSON with a fully local operation from our own machine without even needing a GPU.
19:39It's fast, it's cheap, and most importantly, it's open-source.
19:43So, I encourage you to check it all out.
19:45Behind the scenes, there's different pipelines that use a combination of OCR and layout analysis to structure everything together, put it together as a Pydantic Docling document that then you can use to export to different types of formats,
19:58create data sets, chunk that using the Hybrid Chunkier, and do so much more.
20:03It integrates with a lot of different rag frameworks and agentic systems and harnesses.
20:07So, feel free to try it out.
20:08And I want to give a big thank you to the AI engineering team for having me on.
20:12This is something that I'm really passionate about with processing documents, but also doing this in an open-source way because, you know, we're here at Red Hat and we love open-source.
20:22So, feel free to check out the slides, connect with me on LinkedIn.
20:25I appreciate the opportunity, enjoy the conference, and keep up with the AI engineering ecosystem.
20:31Things are looking super bright right now for the open-source world and for AI engineers in general.
20:37So, see you next time and thanks for watching.