From Ingestion to Agents: How AI Teams Build on Document Intelligence — Adit Abraham, Reducto

AI Engineer · 22 min · 215 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:13Hi.
00:13Can everyone hear me?
00:15Sweet.
00:16We have a short 20 minutes here and so I wanted to jump in and get right to it.
00:21Uh my name is De.
00:22I'm the co-founder and CEO of Redducto.
00:24Uh and today we wanted to talk about one of the I think really practical but maybe less sexy parts of building agents that actually work in the real world
00:34uh which is data.
00:35Um I'm sure you've seen plenty of talks about data today.
00:38We primarily have focused on building infrastructure for anybody working with some of the hardest sources of data which is unstructured images, PDFs, spreadsheets, everything that humans are used to using day-to-day.
00:49Uh, have any of you used Reduct already or trial it?
00:54Cool.
00:55Um, I guess helpful context maybe to start is we're an agentic document processing platform.
01:00Um, so we help a lot of the world's leading AI teams build uh both AI applications and also workflows depending on what they're trying to do.
01:09Uh, that includes a lot of the AI natives that you've probably seen today.
01:12It includes the Harveys, the Lores, the Rogos of the world.
01:15Uh, but also some of the largest enterprises in the world.
01:17And I think that's important context for what we're going to talk about.
01:20Um, we work with the largest tech companies, global financial institutions, insurance orgs, people that have decades worth of historical data that historically has been really, really hard to actually use outside of a demo context.
01:34And across all these companies at this point, we've processed many billions of documents for our customers.
01:39I've gotten kind of lazy with that plus symbol at the end, uh, but the number keeps changing and so we'll let it sit there.
01:46Uh but the main thing that we've learned across those and what I wanted to focus on today is not actually a reduct product itself.
01:52It's the intricacies of what we've learned that hopefully you can actually take home and implement in the work that you're doing as well.
01:59Um and I think a lot of the work that we've done and a lot of the learnings we've had are in this sort of broader thematic context of the scope of AI applications has changed a lot recently.
02:09Uh from just like information synthesis products to actual agents doing work.
02:14And with that we found that there are a few things that are worth talking through.
02:17One is framing the problem itself.
02:18Uh the bottleneck that people face.
02:21Uh why PDFs in particular are hard even though you've probably seen two dozen different PDF processing launches on Twitter.
02:28Uh where we find strengths and weaknesses with different tools.
02:31Uh we think there's a right place in time for traditional CV versus BLMs.
02:35Um and then more interestingly I think the latter half of this talk will actually be focused on the next frontier.
02:41uh what we've seen be possible as a result of having agents in the loop uh things that we found as a result of being able to use harnesses for different types of tasks
02:49uh and our learnings around things like evaluations uh as you go from rag to building agent products but I'll start with the first thing which is something that I assume has been harped on a lot if you went to AI engineer a few years ago
03:02the word that you would have heard in every single talk would have been rag um and everybody was building some form of rag application for a while all applications ations were really some form of information synthesis,
03:13right?
03:14You would pull in information from some context, whether that's a perfect prompt or a file that a user uploaded and you would have something like a search product.
03:22Um, you'd have enterprise search, you'd have a chatbot that would do simple question answer on top of the content and that was it.
03:29But today, the buzzword that you've probably heard a million times is agents.
03:34Uh, and for those of you that are engineers, you've probably had cloud code or similar tool do a lot of your endto-end work for many, many tasks.
03:42And that same sort of shift is starting to happen for all sorts of white collar work.
03:46Um, whether you're in finance or insurance or healthcare, people are starting to make autonomous decisions.
03:52They're trying to create end-to-end work products, not just answer questions from a PDF, but generate and modify PDFs as well.
03:59And that's a very different framing.
04:00the tools that you need, the problems that you face change a lot versus just trying to build a retrieval uh platform.
04:08And the common core for all of this is it actually ends up being an even more importance problem to solve for when you start having these multi-step pipelines.
04:19If you're just doing question answer, there is obviously a risk that comes with just answering the question.
04:25But when agents are making multiple decisions that compound over the course of a source of files, when they're pulling in multiple sources of data, the risk of bad inputs
04:34becomes really, really pervasive in your pipeline.
04:37And that's what we focus on because at the end of the day, a lot of the value of language model tools in the real world only applies in the context to which you apply that intelligence.
04:48And for a lot of enterprises, data is unstructured, it is scattered, and it is multimodal.
04:54uh it's not this like cleanly organized repository.
04:57You have random teams and different organizations that go through and put things in Google Drive and Box and wherever else you might have things.
05:04Um you're going to have data for formats that are unstructured by default.
05:08You don't necessarily know what is in your corpus of information.
05:12And you're going to have all sorts of downstream problems that come with that.
05:15Um some of the problems are going to be parsing extraction accuracy and that definitely matters.
05:19But it's also going to be about problems like are you retrieving the right context?
05:22It's going to be about how do you actually interact with that context and apply modifications to it.
05:28And the thing that you've probably heard harp on and others in the space is PDFs are surprisingly still a very hard problem.
05:36Um I don't know if any of you follow Serge which is a data lab that works with a lot of the foundation model companies.
05:43uh they have this really great benchmark called GDP PDF where even Fable uh like current frontier of model intelligence is at about 30% on their benchmark um and it's entirely predicated on this idea of can we have models go through
05:56and actually make determinations off of contents that would be in documents like PDFs uh and the reason why they're hard I'll come back to that benchmark in a second
06:05is fundamentally PDFs as a file format are both very old but were designed in a very different context I've genuinely met people that have worked on PDF processing longer than I've been alive.
06:16Uh I've met the people that worked on printer drivers for printers to print PDFs in the early 1990s.
06:22And that's what it was like.
06:23You wanted to be able to represent what was faithfully on the documents when you originally created it and have it be printable at the end of it.
06:31A lot of the considerations today are not around that.
06:33Um ultimately what we want is something like a markdown representation, something that agents will effectively reason on.
06:40And in the real world, humans encode so much context visually.
06:44Like the average financial analyst that is a new grad in their IB role uh is not going through and thinking in terms of will agents consume this deck.
06:55They're creating these really creative slides.
06:57I'm sure you've seen the softbank slides with uh goose laying eggs.
07:02Those details matter, right?
07:03Like a lot of the data that you're going to reason on is going to be tabular structures that maybe don't have clean grid lines and separate out the merge cells.
07:10You're going to have things like line charts and graphs.
07:13You're going to have messy handwriting that even I as a human would often struggle to read.
07:18And that's the sort of problem that you need to solve for if you're dealing with that long tail.
07:23And we started the company in 2023 because we felt that there was a sort of step change in what was possible here.
07:30uh for a while as people would think about any sort of PDF processing problem, it used to be some modified version of an NLP pipeline.
07:38You do a simple OCR pass and then you try to post-process that text.
07:42And that worked when you would have really consistent layouts, right?
07:46Like if you knew that a W2 is always going to look like a W2, that's a constrained problem space that you can template your way around.
07:53But VLMs are interesting because they are fundamentally horizontal in nature.
07:57um for the first time you can have this premise of read the document the way that a human would have.
08:02You can have this premise of we want to address the longtail.
08:05Uh and so we found that they were incredible for all sorts of things like handwritten text in a way that traditional OCR just never was.
08:13But the flip side is that we don't think that they're a one-sizefits-all solution.
08:17And if you're solving this problem at scale, if you're a company dealing with hundreds of millions of documents, there are all sorts of secondary considerations like determinism.
08:25you care a lot about the efficiency of your processing.
08:28Uh, and so we find that there are some things where traditional CV is actually still really really strong.
08:34And this has been really underappreciated as autonomous vehicle research has gotten better.
08:38Techniques like object detection are more sophisticated than they were a decade ago.
08:43And so we find that subund million parameter models are really effective for things like detecting the layout of a document.
08:49Um, you can go really really far without even needing a large foundation model.
08:53And these are models that can actually run on CPU.
08:55You can run them at scale.
08:56You can make sure that you understand on a region level what are the hard things for you to process.
09:01And VLMs introduce this notion of semantics.
09:03You can go through and actually identify and make correct the sorts of mistakes that you were likely to have in your pipeline.
09:10And off of that idea of semantics, uh, when you're deconstructing this problem and you have a clear sense of where the more nuanced things are, if you've segmented the text on your page and you understand where the handwriting is, you can introduce this notion of a sort of agent in the loop.
09:25Whereas historically, you would have had a human review team go through and annotate and correct mistakes.
09:30Uh, VLMs can now present this sort of idea of what we call agentic OCR.
09:35Really for us what that looks like is if you've ever used you know a tool like cursor that's applying fast edits in your IDE there's this notion of speculative decoding where you're applying token level edits to your output.
09:46You can apply a similar sort of principle here uh that is not just you know sending OCR to Gemini writing a really pretty prompt asking it nicely to not deviate too much from the original because we find that when you're doing that sort of next token prediction you introduce net new loss cases where models that are really
10:02intelligent will start actually correcting things not faithfully to what was in the document.
10:06They'll see the word total and if the human made a mistake in that table models will actually sometimes go through and add up the values in the table themselves.
10:14What you want is to correct the token level edits that you want.
10:17Maybe you messed up a period versus a comma, a zero versus an O. Those sorts of details really, really matter.
10:22And that's a question of how do we actually represent what a human would have seen if they had read that document.
10:29So the way that we see this is agentic OCR is almost like that human loop analogy where you have the first inputs uh go through with a CD plus VLM parse but then you have a verification
10:41correction layer that ends up leading to a high confidence output.
10:45But I mentioned earlier that we don't see the range of problems as purely just parsing and extraction.
10:51And a lot of what this looks like is I think it's important to think through the details of your pipeline.
10:57Even if you have a great documents to markdown pipeline and a great example of this is if you've built any sort of rag platform um you've probably had some consideration around things like tables.
11:09There are a lot of things that you can encode well in something like markdown.
11:14uh but things like this table where the merge cells actually encode a lot of meaning.
11:19It matters that you're preserving that sort of structure and it's not a model limitation.
11:22LMS are incredible at reasoning through like an HTML structure of the same table, but you're also wasting a lot of tokens and that gets expensive quickly.
11:30And obviously on the other end of the spectrum, you probably don't want to go through and encode simple tables in HTML because then you have a lot of HTML tags that are erroneous.
11:39And so what we ended up doing was looking at this as sort of like a dynamic problem of when you have a simple table, great, we can approximate that data in markdown.
11:48When you have a more complex table, you may want to use something like HTML.
11:52But it's not only language models that should be a consideration in your pipeline.
11:56If you're doing anything related to embedding, you're also going to have the secondary problem of retrieval of that context.
12:02Right?
12:02That same table that I showed you earlier, if you look at the HTML representation is really really messy.
12:09The vast majority of that snippets is just HTML tags.
12:12It's just classifying the structure of the document.
12:15And the unfortunate thing is whereas in some blanket evals, you might have contents that is really trying to do the work for the model and say exactly what you're looking for.
12:26A real world person does not enumerate the values in the table.
12:29They just say how did revenue change over time and they assume that you're going to retrieve the right table when it's relevant.
12:35And whereas language models can reason through that text effectively if you're pulling from a large corpus, we find that embedding models really struggle to correlate that natural language human prompt with this messy blob of HTML tags and numbers.
12:48And so a big thing that you can do that actually takes very little effort is creating a representation that's more so designed for the embedding model itself.
12:56taking the same table, we're creating a natural language representation of that table.
12:59So you have the best of both worlds.
13:01When you're actually passing this into the model for reasoning, you're using the HTML table representation.
13:06And when you're trying to make sure that you retrieve the right snippets, you're using that natural language block.
13:12Off of that, uh there's also this idea of there's a lot to do that is not just parsing and extraction.
13:18Um, and I think a lot of the industry's focus has been on parsing and extraction historically because we do think that there's a massive uplift there.
13:26And I talked earlier about the GDF GDP PDF benchmark which I think is a great illustrative example of what you can see uh as a result of improving your data pipeline.
13:38So what we found is that if you take the same exact benchmark that I mentioned earlier, uh, unfortunately we couldn't test it on Fable because our access was cut.
13:47uh but if you test on other models and you give it both the original PDF but also a structured representation of the PDF like the parse results here across models whether it's Gemini whether it's anthropic or openi
13:59um you find that you actually improve end LLM performance just from better inputs and it's to an extreme where models like GPT 5.5 and opus actually outperform something like Fable out of the box not just on an accuracy basis
14:15but as a result of giving better inputs, the models end up needing to use fewer reasoning tokens as well.
14:21Um, they're focused less on representing the data and more on the actual outputs and as a result they end up driving down latency and also getting to the correct answer more quickly.
14:31But even once you have that sort of pipeline and you've gone through and you've actually inspected everything in your parsing layer, a lot of human work is going to require actually understanding the range of what you have in your corpus,
14:45routing it to the appropriate pipeline and sort of decomposing that problem or even at the end editing and modifying your document.
14:52And so what we tried to do is look at this as this problem of how do you make sure that every interaction that a language model has with a document is as effective as if a human would have done it.
15:01Um if you're filling out a form, how do you make sure that you have precision in where you fill out fields?
15:06And a really good example of this uh on the orchestration side is I think classification splitting are a very underappreciated way to have an LM do its best work.
15:15Obviously, you can just go through and dump as much context as you want.
15:18And if you're doing a sort of needle in the haststack test, that might be fine.
15:22But in practice, there is erosion that you find in quality outside of just the token economics as a result of passing in too much.
15:30And instead, what we find is you can get a lot of headroom by thinking through things like how do you classify the right documents to the right sort of pipeline?
15:39And even for large documents, how do you make sure that you're passing in the snippets uh that are actually relevant?
15:44We see use cases where people will have things like paper mail.
15:48Uh and these paper mail packets can be hundreds of pages long.
15:51You don't necessarily know what is going to be contained within it.
15:54Um you might have issues like a person interle the content and having the model do that sort of work is almost like a distraction from the work that you're actually trying to achieve which might be extracting the data from the paper mail reasoning on it or making a decision.
16:10Again, uh I mentioned earlier that the second half I think is the more interesting piece.
16:15Um which is once you have the sort of initial classification and splitting layer, you've figured out sorting.
16:22Uh I think the thing that we are really excited about as a team is agent harnesses have been this really really interesting frontier to push past what canonically used to be hard unsolved problems.
16:35Um one good example of this that I'll talk about in a second is things like line charts.
16:40uh we work with many of the largest hedge funds in the world and things like line charts historically have been really really difficult because one they're an imaged format but two there's a lot of pixel level granularity that if you're doing anything with a traditional vision encoder
16:53you're probably going to lose you're going to get a rough plot of how revenue trended but you're not going to get the individual data points and so we've been thinking through how do we give agents the ability to have the right tools
17:03to solve for the specific type of problem that you're looking at in the case of chart extraction the chart on the left encodes codes a massive table of data.
17:11If you actually went through and tried to plot every single pixel, it would be really really difficult.
17:16U but it would also be hard for a model to even approximate the intricacies of the lines in between.
17:22And there's no model that out of the box can do this as a singleshot problem.
17:26What you're seeing on the right is a reconstruction of the markdown table that we're able to generate off of the initial line chart.
17:32And the only way that we were able to get there was to have an agent with all sorts of tools.
17:37It has its own code interpreter.
17:38It has the ability to visualize the chart that it's generating.
17:41And it's iteratively going through.
17:42It's finding mistakes in the line chart again and again and again until it's able to get to the final output.
17:48That applies for problems like structured extraction as well where for a while we've had this documents to structured output feature.
17:56Uh but you can really take it a step further by having an agent harness around that same sort of task.
18:01You can have a parent agent go through set validation criteria for sub agents to follow.
18:06Uh and this means that if you have something like a CBP form with tens of thousands of fields, that's the sort of problem where you end up finding a lot of issues that are silent in nature like you drop content, you drop rows and micro one actually released a really good benchmark in the space this morning
18:22where there's this like bifurcation in the market.
18:25Uh Frontier models with Max Reasoning are really really precise.
18:29Like provided that they extracted a row, odds are it's not a hallucinated row like they actually got it correct.
18:34but they silently drop a lot of the contents across the benchmark.
18:37Uh recall really really struggles.
18:39On the flip side, a lot of dedicated document processing services are actually behind frontier models from a precision perspective, but close that gap on a recall perspective.
18:49And so there's always been this sort of trade-off and it was only with an agent harness that we were able to find that sort of local maximum of both
18:56precision and also recall for this sort of task.
19:01The last thing and maybe the most important thing from this talk uh is that I think at the end of the day eval should underpin all of your decisions and it's been a big part of how we think about our product.
19:13Um that applies both to off-the-shelf data sets that you eval against but also to things like real-time production monitoring because your production data is going to differ from whatever else you have in your contrived set.
19:26And I really think it's important to think of eval not as just this like macrolevel view, but also the best teams that we work with look at eval on a granular level for each step of their pipeline.
19:37Uh the first thing might be that you want to make sure that the inputs to your pipeline are great.
19:41And of course, you should eval things like your parsing pipeline.
19:44But even perfect parsing with a horrible retrieval pipeline is not going to help if you're not passing the right context.
19:50And so it's important that you're thinking through details like your retrieval pipeline, your formatting at the end of the pipeline, and also ultimately the most important thing is are you able to improve end agent performance.
20:03I'll close off just with a a sense of where we are headed and where we've seen the industry head.
20:09Uh the most important thing I think is as agents get better and better, you can deviate from the sort of deterministic pipeline that you would have had a few years ago.
20:18Um, a lot of our customers will actually create effectively a file system for their agent to go through and navigate and let the agent decide what sorts of tools it wants to use.
20:26So, we create a CLI where instead of people creating a endto-end pipeline where documents always follow one specific flow, um, the agent will decide if it needs to read a certain type of document and they'll split that into two sets.
20:39Uh, one is a content field which the agent can read as it needs to.
20:42Uh, but the other is all the metadata that you would have wanted.
20:45If you're doing things like citations, you may want bounding boxes, so on and so forth.
20:51Um, I'll skip this part on editing.
20:53I think there's a lot of interesting work being done here.
20:55We've already released some of it, but in the next few months, you will see us look more and more towards things like document generation.
21:02But the recap for today, and I really appreciate your time, is one, I highly recommend that you decompose the parsing problem.
21:09To the extent possible, you should think of it as the right tool for the right task so that you can hit that perimeter frontier of accuracy, cost, and latency.
21:17Two, I think agentic verification is the biggest step change that the industry has had for a while and it's a really good opportunity for you to make sure that you're building pipelines that work in production.
21:27Three, I think it takes very little effort, but there's a lot of headroom from details like formatting the data for its consumer.
21:35or similar vein, I think it's really really important to think about not just the data processing but the the data orchestration and so you should always think about tools like classify and split as a way to augment your pipeline.
21:47Five uh make sure that you eval at every stage and six think through what that next frontier looks like for you because I think most successful companies in today's era have deviated a lot from what we used to do two three years ago.
22:00But if you have any questions uh please feel free to reach out at any point.
22:04My email is just first namered reductto.ai.
22:07Uh and you can also reach out on our website if we can be helpful for your use case.
22:10Thank you.