WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=0I07YAuF8xA NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/0I07YAuF8xA.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:01.309 --> 00:00:03.309 [music] s2 00:00:13.040 --> 00:00:13.599 Hi. s3 00:00:13.599 --> 00:00:15.599 Can everyone hear me? s4 00:00:15.599 --> 00:00:16.400 Sweet. s5 00:00:16.400 --> 00:00:21.119 We have a short 20 minutes here and so I wanted to jump in and get right to it. s6 00:00:21.119 --> 00:00:22.160 Uh my name is De. s7 00:00:22.160 --> 00:00:24.240 I'm the co-founder and CEO of Redducto. s8 00:00:24.240 --> 00:00:34.239 Uh and today we wanted to talk about one of the I think really practical but maybe less sexy parts of building agents that actually work in the real world s9 00:00:34.239 --> 00:00:35.200 uh which is data. s10 00:00:35.200 --> 00:00:38.399 Um I'm sure you've seen plenty of talks about data today. s11 00:00:38.399 --> 00:00:49.840 We primarily have focused on building infrastructure for anybody working with some of the hardest sources of data which is unstructured images, PDFs, spreadsheets, everything that humans are used to using day-to-day. s12 00:00:49.840 --> 00:00:54.800 Uh, have any of you used Reduct already or trial it? s13 00:00:54.800 --> 00:00:55.520 Cool. s14 00:00:55.520 --> 00:01:00.960 Um, I guess helpful context maybe to start is we're an agentic document processing platform. s15 00:01:00.960 --> 00:01:09.119 Um, so we help a lot of the world's leading AI teams build uh both AI applications and also workflows depending on what they're trying to do. s16 00:01:09.119 --> 00:01:12.159 Uh, that includes a lot of the AI natives that you've probably seen today. s17 00:01:12.159 --> 00:01:15.119 It includes the Harveys, the Lores, the Rogos of the world. s18 00:01:15.119 --> 00:01:17.520 Uh, but also some of the largest enterprises in the world. s19 00:01:17.520 --> 00:01:20.240 And I think that's important context for what we're going to talk about. s20 00:01:20.240 --> 00:01:34.479 Um, we work with the largest tech companies, global financial institutions, insurance orgs, people that have decades worth of historical data that historically has been really, really hard to actually use outside of a demo context. s21 00:01:34.479 --> 00:01:39.920 And across all these companies at this point, we've processed many billions of documents for our customers. s22 00:01:39.920 --> 00:01:46.240 I've gotten kind of lazy with that plus symbol at the end, uh, but the number keeps changing and so we'll let it sit there. s23 00:01:46.240 --> 00:01:52.720 Uh but the main thing that we've learned across those and what I wanted to focus on today is not actually a reduct product itself. s24 00:01:52.720 --> 00:01:59.840 It's the intricacies of what we've learned that hopefully you can actually take home and implement in the work that you're doing as well. s25 00:01:59.840 --> 00:02:09.200 Um and I think a lot of the work that we've done and a lot of the learnings we've had are in this sort of broader thematic context of the scope of AI applications has changed a lot recently. s26 00:02:09.200 --> 00:02:14.319 Uh from just like information synthesis products to actual agents doing work. s27 00:02:14.319 --> 00:02:17.520 And with that we found that there are a few things that are worth talking through. s28 00:02:17.520 --> 00:02:18.959 One is framing the problem itself. s29 00:02:18.959 --> 00:02:21.040 Uh the bottleneck that people face. s30 00:02:21.040 --> 00:02:28.640 Uh why PDFs in particular are hard even though you've probably seen two dozen different PDF processing launches on Twitter. s31 00:02:28.640 --> 00:02:31.680 Uh where we find strengths and weaknesses with different tools. s32 00:02:31.680 --> 00:02:35.840 Uh we think there's a right place in time for traditional CV versus BLMs. s33 00:02:35.840 --> 00:02:41.280 Um and then more interestingly I think the latter half of this talk will actually be focused on the next frontier. s34 00:02:41.280 --> 00:02:49.120 uh what we've seen be possible as a result of having agents in the loop uh things that we found as a result of being able to use harnesses for different types of tasks s35 00:02:49.120 --> 00:03:02.159 uh and our learnings around things like evaluations uh as you go from rag to building agent products but I'll start with the first thing which is something that I assume has been harped on a lot if you went to AI engineer a few years ago s36 00:03:02.159 --> 00:03:13.360 the word that you would have heard in every single talk would have been rag um and everybody was building some form of rag application for a while all applications ations were really some form of information synthesis, s37 00:03:13.360 --> 00:03:14.000 right? s38 00:03:14.000 --> 00:03:22.640 You would pull in information from some context, whether that's a perfect prompt or a file that a user uploaded and you would have something like a search product. s39 00:03:22.640 --> 00:03:29.920 Um, you'd have enterprise search, you'd have a chatbot that would do simple question answer on top of the content and that was it. s40 00:03:29.920 --> 00:03:34.799 But today, the buzzword that you've probably heard a million times is agents. s41 00:03:34.799 --> 00:03:42.879 Uh, and for those of you that are engineers, you've probably had cloud code or similar tool do a lot of your endto-end work for many, many tasks. s42 00:03:42.879 --> 00:03:46.799 And that same sort of shift is starting to happen for all sorts of white collar work. s43 00:03:46.799 --> 00:03:52.239 Um, whether you're in finance or insurance or healthcare, people are starting to make autonomous decisions. s44 00:03:52.239 --> 00:03:59.120 They're trying to create end-to-end work products, not just answer questions from a PDF, but generate and modify PDFs as well. s45 00:03:59.120 --> 00:04:00.560 And that's a very different framing. s46 00:04:00.560 --> 00:04:08.319 the tools that you need, the problems that you face change a lot versus just trying to build a retrieval uh platform. s47 00:04:08.319 --> 00:04:19.120 And the common core for all of this is it actually ends up being an even more importance problem to solve for when you start having these multi-step pipelines. s48 00:04:19.120 --> 00:04:25.280 If you're just doing question answer, there is obviously a risk that comes with just answering the question. s49 00:04:25.280 --> 00:04:34.240 But when agents are making multiple decisions that compound over the course of a source of files, when they're pulling in multiple sources of data, the risk of bad inputs s50 00:04:34.240 --> 00:04:37.360 becomes really, really pervasive in your pipeline. s51 00:04:37.360 --> 00:04:48.400 And that's what we focus on because at the end of the day, a lot of the value of language model tools in the real world only applies in the context to which you apply that intelligence. s52 00:04:48.400 --> 00:04:54.240 And for a lot of enterprises, data is unstructured, it is scattered, and it is multimodal. s53 00:04:54.240 --> 00:04:57.520 uh it's not this like cleanly organized repository. s54 00:04:57.520 --> 00:05:04.800 You have random teams and different organizations that go through and put things in Google Drive and Box and wherever else you might have things. s55 00:05:04.800 --> 00:05:08.479 Um you're going to have data for formats that are unstructured by default. s56 00:05:08.479 --> 00:05:12.160 You don't necessarily know what is in your corpus of information. s57 00:05:12.160 --> 00:05:15.120 And you're going to have all sorts of downstream problems that come with that. s58 00:05:15.120 --> 00:05:19.600 Um some of the problems are going to be parsing extraction accuracy and that definitely matters. s59 00:05:19.600 --> 00:05:22.720 But it's also going to be about problems like are you retrieving the right context? s60 00:05:22.720 --> 00:05:28.720 It's going to be about how do you actually interact with that context and apply modifications to it. s61 00:05:28.720 --> 00:05:36.639 And the thing that you've probably heard harp on and others in the space is PDFs are surprisingly still a very hard problem. s62 00:05:36.639 --> 00:05:43.039 Um I don't know if any of you follow Serge which is a data lab that works with a lot of the foundation model companies. s63 00:05:43.039 --> 00:05:56.639 uh they have this really great benchmark called GDP PDF where even Fable uh like current frontier of model intelligence is at about 30% on their benchmark um and it's entirely predicated on this idea of can we have models go through s64 00:05:56.639 --> 00:06:05.440 and actually make determinations off of contents that would be in documents like PDFs uh and the reason why they're hard I'll come back to that benchmark in a second s65 00:06:05.440 --> 00:06:16.800 is fundamentally PDFs as a file format are both very old but were designed in a very different context I've genuinely met people that have worked on PDF processing longer than I've been alive. s66 00:06:16.800 --> 00:06:22.319 Uh I've met the people that worked on printer drivers for printers to print PDFs in the early 1990s. s67 00:06:22.319 --> 00:06:23.360 And that's what it was like. s68 00:06:23.360 --> 00:06:31.039 You wanted to be able to represent what was faithfully on the documents when you originally created it and have it be printable at the end of it. s69 00:06:31.039 --> 00:06:33.520 A lot of the considerations today are not around that. s70 00:06:33.520 --> 00:06:40.400 Um ultimately what we want is something like a markdown representation, something that agents will effectively reason on. s71 00:06:40.400 --> 00:06:44.960 And in the real world, humans encode so much context visually. s72 00:06:44.960 --> 00:06:55.280 Like the average financial analyst that is a new grad in their IB role uh is not going through and thinking in terms of will agents consume this deck. s73 00:06:55.280 --> 00:06:57.759 They're creating these really creative slides. s74 00:06:57.759 --> 00:07:02.000 I'm sure you've seen the softbank slides with uh goose laying eggs. s75 00:07:02.000 --> 00:07:03.440 Those details matter, right? s76 00:07:03.440 --> 00:07:10.800 Like a lot of the data that you're going to reason on is going to be tabular structures that maybe don't have clean grid lines and separate out the merge cells. s77 00:07:10.800 --> 00:07:13.199 You're going to have things like line charts and graphs. s78 00:07:13.199 --> 00:07:18.000 You're going to have messy handwriting that even I as a human would often struggle to read. s79 00:07:18.000 --> 00:07:23.599 And that's the sort of problem that you need to solve for if you're dealing with that long tail. s80 00:07:23.599 --> 00:07:30.479 And we started the company in 2023 because we felt that there was a sort of step change in what was possible here. s81 00:07:30.479 --> 00:07:38.800 uh for a while as people would think about any sort of PDF processing problem, it used to be some modified version of an NLP pipeline. s82 00:07:38.800 --> 00:07:42.720 You do a simple OCR pass and then you try to post-process that text. s83 00:07:42.720 --> 00:07:46.160 And that worked when you would have really consistent layouts, right? s84 00:07:46.160 --> 00:07:53.120 Like if you knew that a W2 is always going to look like a W2, that's a constrained problem space that you can template your way around. s85 00:07:53.120 --> 00:07:57.360 But VLMs are interesting because they are fundamentally horizontal in nature. s86 00:07:57.360 --> 00:08:02.639 um for the first time you can have this premise of read the document the way that a human would have. s87 00:08:02.639 --> 00:08:05.680 You can have this premise of we want to address the longtail. s88 00:08:05.680 --> 00:08:13.520 Uh and so we found that they were incredible for all sorts of things like handwritten text in a way that traditional OCR just never was. s89 00:08:13.520 --> 00:08:17.360 But the flip side is that we don't think that they're a one-sizefits-all solution. s90 00:08:17.360 --> 00:08:25.759 And if you're solving this problem at scale, if you're a company dealing with hundreds of millions of documents, there are all sorts of secondary considerations like determinism. s91 00:08:25.759 --> 00:08:28.639 you care a lot about the efficiency of your processing. s92 00:08:28.639 --> 00:08:34.240 Uh, and so we find that there are some things where traditional CV is actually still really really strong. s93 00:08:34.240 --> 00:08:38.560 And this has been really underappreciated as autonomous vehicle research has gotten better. s94 00:08:38.560 --> 00:08:43.279 Techniques like object detection are more sophisticated than they were a decade ago. s95 00:08:43.279 --> 00:08:49.200 And so we find that subund million parameter models are really effective for things like detecting the layout of a document. s96 00:08:49.200 --> 00:08:53.360 Um, you can go really really far without even needing a large foundation model. s97 00:08:53.360 --> 00:08:55.600 And these are models that can actually run on CPU. s98 00:08:55.600 --> 00:08:56.720 You can run them at scale. s99 00:08:56.720 --> 00:09:01.680 You can make sure that you understand on a region level what are the hard things for you to process. s100 00:09:01.680 --> 00:09:03.839 And VLMs introduce this notion of semantics. s101 00:09:03.839 --> 00:09:10.560 You can go through and actually identify and make correct the sorts of mistakes that you were likely to have in your pipeline. s102 00:09:10.720 --> 00:09:25.920 And off of that idea of semantics, uh, when you're deconstructing this problem and you have a clear sense of where the more nuanced things are, if you've segmented the text on your page and you understand where the handwriting is, you can introduce this notion of a sort of agent in the loop. s103 00:09:25.920 --> 00:09:30.399 Whereas historically, you would have had a human review team go through and annotate and correct mistakes. s104 00:09:30.399 --> 00:09:35.120 Uh, VLMs can now present this sort of idea of what we call agentic OCR. s105 00:09:35.120 --> 00:09:46.240 Really for us what that looks like is if you've ever used you know a tool like cursor that's applying fast edits in your IDE there's this notion of speculative decoding where you're applying token level edits to your output. s106 00:09:46.240 --> 00:10:02.480 You can apply a similar sort of principle here uh that is not just you know sending OCR to Gemini writing a really pretty prompt asking it nicely to not deviate too much from the original because we find that when you're doing that sort of next token prediction you introduce net new loss cases where models that are really s107 00:10:02.480 --> 00:10:06.480 intelligent will start actually correcting things not faithfully to what was in the document. s108 00:10:06.480 --> 00:10:14.480 They'll see the word total and if the human made a mistake in that table models will actually sometimes go through and add up the values in the table themselves. s109 00:10:14.480 --> 00:10:17.279 What you want is to correct the token level edits that you want. s110 00:10:17.279 --> 00:10:22.800 Maybe you messed up a period versus a comma, a zero versus an O. Those sorts of details really, really matter. s111 00:10:22.800 --> 00:10:28.959 And that's a question of how do we actually represent what a human would have seen if they had read that document. s112 00:10:29.120 --> 00:10:41.120 So the way that we see this is agentic OCR is almost like that human loop analogy where you have the first inputs uh go through with a CD plus VLM parse but then you have a verification s113 00:10:41.120 --> 00:10:45.680 correction layer that ends up leading to a high confidence output. s114 00:10:45.680 --> 00:10:51.760 But I mentioned earlier that we don't see the range of problems as purely just parsing and extraction. s115 00:10:51.760 --> 00:10:57.760 And a lot of what this looks like is I think it's important to think through the details of your pipeline. s116 00:10:57.760 --> 00:11:09.760 Even if you have a great documents to markdown pipeline and a great example of this is if you've built any sort of rag platform um you've probably had some consideration around things like tables. s117 00:11:09.760 --> 00:11:14.000 There are a lot of things that you can encode well in something like markdown. s118 00:11:14.000 --> 00:11:19.200 uh but things like this table where the merge cells actually encode a lot of meaning. s119 00:11:19.200 --> 00:11:22.800 It matters that you're preserving that sort of structure and it's not a model limitation. s120 00:11:22.800 --> 00:11:30.480 LMS are incredible at reasoning through like an HTML structure of the same table, but you're also wasting a lot of tokens and that gets expensive quickly. s121 00:11:30.480 --> 00:11:39.440 And obviously on the other end of the spectrum, you probably don't want to go through and encode simple tables in HTML because then you have a lot of HTML tags that are erroneous. s122 00:11:39.440 --> 00:11:48.240 And so what we ended up doing was looking at this as sort of like a dynamic problem of when you have a simple table, great, we can approximate that data in markdown. s123 00:11:48.240 --> 00:11:52.079 When you have a more complex table, you may want to use something like HTML. s124 00:11:52.079 --> 00:11:56.160 But it's not only language models that should be a consideration in your pipeline. s125 00:11:56.160 --> 00:12:02.320 If you're doing anything related to embedding, you're also going to have the secondary problem of retrieval of that context. s126 00:12:02.320 --> 00:12:02.720 Right? s127 00:12:02.720 --> 00:12:09.360 That same table that I showed you earlier, if you look at the HTML representation is really really messy. s128 00:12:09.360 --> 00:12:12.720 The vast majority of that snippets is just HTML tags. s129 00:12:12.720 --> 00:12:15.360 It's just classifying the structure of the document. s130 00:12:15.360 --> 00:12:26.160 And the unfortunate thing is whereas in some blanket evals, you might have contents that is really trying to do the work for the model and say exactly what you're looking for. s131 00:12:26.160 --> 00:12:29.360 A real world person does not enumerate the values in the table. s132 00:12:29.360 --> 00:12:35.279 They just say how did revenue change over time and they assume that you're going to retrieve the right table when it's relevant. s133 00:12:35.279 --> 00:12:48.560 And whereas language models can reason through that text effectively if you're pulling from a large corpus, we find that embedding models really struggle to correlate that natural language human prompt with this messy blob of HTML tags and numbers. s134 00:12:48.560 --> 00:12:56.000 And so a big thing that you can do that actually takes very little effort is creating a representation that's more so designed for the embedding model itself. s135 00:12:56.000 --> 00:12:59.839 taking the same table, we're creating a natural language representation of that table. s136 00:12:59.839 --> 00:13:01.120 So you have the best of both worlds. s137 00:13:01.120 --> 00:13:06.079 When you're actually passing this into the model for reasoning, you're using the HTML table representation. s138 00:13:06.079 --> 00:13:12.079 And when you're trying to make sure that you retrieve the right snippets, you're using that natural language block. s139 00:13:12.720 --> 00:13:18.880 Off of that, uh there's also this idea of there's a lot to do that is not just parsing and extraction. s140 00:13:18.880 --> 00:13:26.320 Um, and I think a lot of the industry's focus has been on parsing and extraction historically because we do think that there's a massive uplift there. s141 00:13:26.320 --> 00:13:38.639 And I talked earlier about the GDF GDP PDF benchmark which I think is a great illustrative example of what you can see uh as a result of improving your data pipeline. s142 00:13:38.639 --> 00:13:47.040 So what we found is that if you take the same exact benchmark that I mentioned earlier, uh, unfortunately we couldn't test it on Fable because our access was cut. s143 00:13:47.040 --> 00:13:59.680 uh but if you test on other models and you give it both the original PDF but also a structured representation of the PDF like the parse results here across models whether it's Gemini whether it's anthropic or openi s144 00:13:59.680 --> 00:14:15.680 um you find that you actually improve end LLM performance just from better inputs and it's to an extreme where models like GPT 5.5 and opus actually outperform something like Fable out of the box not just on an accuracy basis s145 00:14:15.680 --> 00:14:21.440 but as a result of giving better inputs, the models end up needing to use fewer reasoning tokens as well. s146 00:14:21.440 --> 00:14:31.680 Um, they're focused less on representing the data and more on the actual outputs and as a result they end up driving down latency and also getting to the correct answer more quickly. s147 00:14:31.680 --> 00:14:45.920 But even once you have that sort of pipeline and you've gone through and you've actually inspected everything in your parsing layer, a lot of human work is going to require actually understanding the range of what you have in your corpus, s148 00:14:45.920 --> 00:14:52.800 routing it to the appropriate pipeline and sort of decomposing that problem or even at the end editing and modifying your document. s149 00:14:52.800 --> 00:15:01.360 And so what we tried to do is look at this as this problem of how do you make sure that every interaction that a language model has with a document is as effective as if a human would have done it. s150 00:15:01.360 --> 00:15:06.320 Um if you're filling out a form, how do you make sure that you have precision in where you fill out fields? s151 00:15:06.320 --> 00:15:15.600 And a really good example of this uh on the orchestration side is I think classification splitting are a very underappreciated way to have an LM do its best work. s152 00:15:15.600 --> 00:15:18.800 Obviously, you can just go through and dump as much context as you want. s153 00:15:18.800 --> 00:15:22.079 And if you're doing a sort of needle in the haststack test, that might be fine. s154 00:15:22.079 --> 00:15:30.000 But in practice, there is erosion that you find in quality outside of just the token economics as a result of passing in too much. s155 00:15:30.000 --> 00:15:39.279 And instead, what we find is you can get a lot of headroom by thinking through things like how do you classify the right documents to the right sort of pipeline? s156 00:15:39.279 --> 00:15:44.800 And even for large documents, how do you make sure that you're passing in the snippets uh that are actually relevant? s157 00:15:44.800 --> 00:15:48.160 We see use cases where people will have things like paper mail. s158 00:15:48.160 --> 00:15:51.680 Uh and these paper mail packets can be hundreds of pages long. s159 00:15:51.680 --> 00:15:54.399 You don't necessarily know what is going to be contained within it. s160 00:15:54.399 --> 00:16:09.680 Um you might have issues like a person interle the content and having the model do that sort of work is almost like a distraction from the work that you're actually trying to achieve which might be extracting the data from the paper mail reasoning on it or making a decision. s161 00:16:10.639 --> 00:16:15.839 Again, uh I mentioned earlier that the second half I think is the more interesting piece. s162 00:16:15.839 --> 00:16:22.720 Um which is once you have the sort of initial classification and splitting layer, you've figured out sorting. s163 00:16:22.720 --> 00:16:35.839 Uh I think the thing that we are really excited about as a team is agent harnesses have been this really really interesting frontier to push past what canonically used to be hard unsolved problems. s164 00:16:35.839 --> 00:16:40.320 Um one good example of this that I'll talk about in a second is things like line charts. s165 00:16:40.320 --> 00:16:53.279 uh we work with many of the largest hedge funds in the world and things like line charts historically have been really really difficult because one they're an imaged format but two there's a lot of pixel level granularity that if you're doing anything with a traditional vision encoder s166 00:16:53.279 --> 00:17:03.040 you're probably going to lose you're going to get a rough plot of how revenue trended but you're not going to get the individual data points and so we've been thinking through how do we give agents the ability to have the right tools s167 00:17:03.040 --> 00:17:11.919 to solve for the specific type of problem that you're looking at in the case of chart extraction the chart on the left encodes codes a massive table of data. s168 00:17:11.919 --> 00:17:16.480 If you actually went through and tried to plot every single pixel, it would be really really difficult. s169 00:17:16.480 --> 00:17:22.720 U but it would also be hard for a model to even approximate the intricacies of the lines in between. s170 00:17:22.720 --> 00:17:26.160 And there's no model that out of the box can do this as a singleshot problem. s171 00:17:26.160 --> 00:17:32.960 What you're seeing on the right is a reconstruction of the markdown table that we're able to generate off of the initial line chart. s172 00:17:32.960 --> 00:17:37.200 And the only way that we were able to get there was to have an agent with all sorts of tools. s173 00:17:37.200 --> 00:17:38.559 It has its own code interpreter. s174 00:17:38.559 --> 00:17:41.200 It has the ability to visualize the chart that it's generating. s175 00:17:41.200 --> 00:17:42.400 And it's iteratively going through. s176 00:17:42.400 --> 00:17:48.720 It's finding mistakes in the line chart again and again and again until it's able to get to the final output. s177 00:17:48.720 --> 00:17:56.240 That applies for problems like structured extraction as well where for a while we've had this documents to structured output feature. s178 00:17:56.240 --> 00:18:01.120 Uh but you can really take it a step further by having an agent harness around that same sort of task. s179 00:18:01.120 --> 00:18:06.480 You can have a parent agent go through set validation criteria for sub agents to follow. s180 00:18:06.480 --> 00:18:22.799 Uh and this means that if you have something like a CBP form with tens of thousands of fields, that's the sort of problem where you end up finding a lot of issues that are silent in nature like you drop content, you drop rows and micro one actually released a really good benchmark in the space this morning s181 00:18:22.799 --> 00:18:25.520 where there's this like bifurcation in the market. s182 00:18:25.520 --> 00:18:29.200 Uh Frontier models with Max Reasoning are really really precise. s183 00:18:29.200 --> 00:18:34.400 Like provided that they extracted a row, odds are it's not a hallucinated row like they actually got it correct. s184 00:18:34.400 --> 00:18:37.600 but they silently drop a lot of the contents across the benchmark. s185 00:18:37.600 --> 00:18:39.840 Uh recall really really struggles. s186 00:18:39.840 --> 00:18:49.600 On the flip side, a lot of dedicated document processing services are actually behind frontier models from a precision perspective, but close that gap on a recall perspective. s187 00:18:49.600 --> 00:18:56.880 And so there's always been this sort of trade-off and it was only with an agent harness that we were able to find that sort of local maximum of both s188 00:18:56.880 --> 00:19:01.200 precision and also recall for this sort of task. s189 00:19:01.360 --> 00:19:13.600 The last thing and maybe the most important thing from this talk uh is that I think at the end of the day eval should underpin all of your decisions and it's been a big part of how we think about our product. s190 00:19:13.600 --> 00:19:26.320 Um that applies both to off-the-shelf data sets that you eval against but also to things like real-time production monitoring because your production data is going to differ from whatever else you have in your contrived set. s191 00:19:26.320 --> 00:19:37.280 And I really think it's important to think of eval not as just this like macrolevel view, but also the best teams that we work with look at eval on a granular level for each step of their pipeline. s192 00:19:37.280 --> 00:19:41.600 Uh the first thing might be that you want to make sure that the inputs to your pipeline are great. s193 00:19:41.600 --> 00:19:44.880 And of course, you should eval things like your parsing pipeline. s194 00:19:44.880 --> 00:19:50.799 But even perfect parsing with a horrible retrieval pipeline is not going to help if you're not passing the right context. s195 00:19:50.799 --> 00:20:03.520 And so it's important that you're thinking through details like your retrieval pipeline, your formatting at the end of the pipeline, and also ultimately the most important thing is are you able to improve end agent performance. s196 00:20:03.919 --> 00:20:09.200 I'll close off just with a a sense of where we are headed and where we've seen the industry head. s197 00:20:09.200 --> 00:20:18.000 Uh the most important thing I think is as agents get better and better, you can deviate from the sort of deterministic pipeline that you would have had a few years ago. s198 00:20:18.000 --> 00:20:26.640 Um, a lot of our customers will actually create effectively a file system for their agent to go through and navigate and let the agent decide what sorts of tools it wants to use. s199 00:20:26.640 --> 00:20:39.039 So, we create a CLI where instead of people creating a endto-end pipeline where documents always follow one specific flow, um, the agent will decide if it needs to read a certain type of document and they'll split that into two sets. s200 00:20:39.039 --> 00:20:42.880 Uh, one is a content field which the agent can read as it needs to. s201 00:20:42.880 --> 00:20:45.760 Uh, but the other is all the metadata that you would have wanted. s202 00:20:45.760 --> 00:20:51.360 If you're doing things like citations, you may want bounding boxes, so on and so forth. s203 00:20:51.360 --> 00:20:53.360 Um, I'll skip this part on editing. s204 00:20:53.360 --> 00:20:55.520 I think there's a lot of interesting work being done here. s205 00:20:55.520 --> 00:21:02.559 We've already released some of it, but in the next few months, you will see us look more and more towards things like document generation. s206 00:21:02.559 --> 00:21:09.919 But the recap for today, and I really appreciate your time, is one, I highly recommend that you decompose the parsing problem. s207 00:21:09.919 --> 00:21:17.200 To the extent possible, you should think of it as the right tool for the right task so that you can hit that perimeter frontier of accuracy, cost, and latency. s208 00:21:17.200 --> 00:21:27.600 Two, I think agentic verification is the biggest step change that the industry has had for a while and it's a really good opportunity for you to make sure that you're building pipelines that work in production. s209 00:21:27.600 --> 00:21:35.440 Three, I think it takes very little effort, but there's a lot of headroom from details like formatting the data for its consumer. s210 00:21:35.440 --> 00:21:47.679 or similar vein, I think it's really really important to think about not just the data processing but the the data orchestration and so you should always think about tools like classify and split as a way to augment your pipeline. s211 00:21:47.679 --> 00:22:00.480 Five uh make sure that you eval at every stage and six think through what that next frontier looks like for you because I think most successful companies in today's era have deviated a lot from what we used to do two three years ago. s212 00:22:00.480 --> 00:22:04.159 But if you have any questions uh please feel free to reach out at any point. s213 00:22:04.159 --> 00:22:07.280 My email is just first namered reductto.ai. s214 00:22:07.280 --> 00:22:10.880 Uh and you can also reach out on our website if we can be helpful for your use case. s215 00:22:10.880 --> 00:22:13.039 Thank you.