WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=_gvamfT8H-w NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/_gvamfT8H-w.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:01.309 --> 00:00:03.309 [music] s2 00:00:13.519 --> 00:00:16.080 Uh, welcome everyone to our session. s3 00:00:16.080 --> 00:00:17.039 Um, I'm Haral. s4 00:00:17.039 --> 00:00:21.520 I'm a senior director of products um at DocYsine and I'm joined by Sean. s5 00:00:21.520 --> 00:00:22.080 Hello everyone. s6 00:00:22.080 --> 00:00:23.840 I'm a product manager at NVIDIA. s7 00:00:23.840 --> 00:00:34.160 So today Sean and I are going to talk about a massive problem that every enterprise faces which is agreement data largecale agreement data. s8 00:00:34.160 --> 00:00:53.120 Agreements are a big part of any relationship any B-2B organization kind of goes through day in day out and a lot of that data is captured inside that agreements and it's very critical whether it's pricing tables a lot of it and it's all in a lot of different unstructured s9 00:00:53.120 --> 00:01:07.040 format and that's kind of what we're going to show is how docysine partner with Nvidia are fixing that on making that data available readable usable for a lot of our kind of organizations. s10 00:01:07.040 --> 00:01:10.080 So here is kind of just a quick map of our talk today. s11 00:01:10.080 --> 00:01:11.520 We'll start with just the stakes. s12 00:01:11.520 --> 00:01:12.880 Why does this matter? s13 00:01:12.880 --> 00:01:15.200 Why the scale is so large? s14 00:01:15.200 --> 00:01:25.040 And then we'll dive deep into the technical architecture of how we are approaching it, how we have tackled this thing, especially this document processing at at scale. s15 00:01:25.040 --> 00:01:33.119 And then finally, we'll cover what we have learned from our evaluation of all of the different models we've tried for different purposes. s16 00:01:33.119 --> 00:01:35.680 and share our learnings with you. s17 00:01:35.680 --> 00:01:49.920 So why like you know just to understand why this is a big problem when you think about docysine raise your hands how many of you have used docuign anyone who's employed probably the HR docs right so it's a massive s18 00:01:49.920 --> 00:02:03.520 scale everyone uses docy for us it's a massive engineering problem as well because just look at the scale we have 1.9 million customers who are paying us and a billion users s19 00:02:03.520 --> 00:02:19.680 what does that imply we process us a million agreements a day that needs to now structurize make it readable make it querable usable and you know in the past we've worked with Deote on a study and it says that there's $2 trillion s20 00:02:19.680 --> 00:02:32.879 captured in this agreement negotiated value that no one capitalizes no one goes back and gets that um data back right and why it's because they have to do a lot of human reading human reviews there's disconnected s21 00:02:32.879 --> 00:02:36.239 systems lot of manual workflow flows that are there. s22 00:02:36.239 --> 00:02:49.440 So that's kind of why docysine built AM an intelligent agreement management platform that takes the entire like applying in an AI first way the entire agreement life cycle whether you are creating agreements s23 00:02:49.440 --> 00:03:00.560 genai helps a lot with that whether you're negotiating to understanding and redlinining all the way to after signing storing and making a lot of insights from this data. s24 00:03:00.560 --> 00:03:17.440 So when you think about the challenges involved right in an agreement there is the unstructured data it could be a PDF it could be a PNG and what are people wanting to do is simple questions they can't get that that is the data that is trapped inside s25 00:03:17.440 --> 00:03:25.840 one agreement but also the whole corpus of millions of agreement 10 years 20 years of business has kind of put into that. s26 00:03:25.840 --> 00:03:37.280 So for agreements aren't flat they're also hierarchical in nature they are like you know one agreement governs the other the other kind of does something else so you always are needing s27 00:03:37.280 --> 00:03:50.000 lot of things to answer this question simple thing I'm sure you all are using a lot of right claude and all especially at your company's organization simple thing what did we contract for the total tokens right with claude no one knows that's captured s28 00:03:50.000 --> 00:04:00.959 inside this agreement in different forms and fashion so you need to be extracting this data to find things but also insights and push it downstream where you're tracking doing more things s29 00:04:00.959 --> 00:04:14.000 and when you analyze an enterprise contract a big set of things are captured what I call vital terms like pricing tiers the skues the information SLAs's rate cards they're all in table format s30 00:04:14.000 --> 00:04:28.639 now traditional document extraction tools or a generic VM BLM completely fail here like you know we've tried we've definitely done this because they're reading text line by line which breaks a lot of that concept within the table. s31 00:04:28.639 --> 00:04:31.600 A merge sells some things are not boundaried. s32 00:04:31.600 --> 00:04:43.520 So this makes a massive operational overhead for the downstream legal teams, procurement team, sales teams to get that queries and get that answers done and they spend hours and hours digging through this s33 00:04:43.520 --> 00:04:45.600 um to even just locate a basic thing. s34 00:04:45.600 --> 00:04:57.840 And that's kind of where we partnered with Nvidia and leveraged a purpose-built model like you know tool which is literally like you know making our architecture for table extraction s35 00:04:57.840 --> 00:05:00.960 take us to make and solve these complex use cases. s36 00:05:00.960 --> 00:05:04.080 We're trying with this we're making things scalable. s37 00:05:04.080 --> 00:05:10.240 We can understand it with the layout but also deliver really accurate results. s38 00:05:10.240 --> 00:05:15.600 And to share more about how we're leveraging the Neotron, I'm going to hand it to Sean. s39 00:05:17.120 --> 00:05:17.440 All right. s40 00:05:17.440 --> 00:05:18.240 Hello everyone. s41 00:05:18.240 --> 00:05:22.479 Uh so real quick on uh the Neotron retriever initiative. s42 00:05:22.479 --> 00:05:26.400 So for those who here knows about Neotron, you raise your hand real quick. s43 00:05:26.400 --> 00:05:27.120 Awesome. s44 00:05:27.120 --> 00:05:41.039 Uh so Neimatron is all about building worldclass open-source models and publishing the data sets, the techniques, uh the quantization approaches, distillation approaches, pruning approaches, every technique possible, blueprints to go with that, you name it. s45 00:05:41.039 --> 00:05:50.160 Um throughout the Neimatron portfolio, we have specifically Neatron Retriever, which is building embedding models, reranking models, and document extraction models. s46 00:05:50.160 --> 00:06:03.120 Um so real quick here we sort of our first initiative is if you're a large scale enterprise that deals with pabytes scale data our first initiative is how do we make sure that you find the right document given a certain query your agent sends s47 00:06:03.120 --> 00:06:05.759 you know set of queries to the corpus afterwards. s48 00:06:05.759 --> 00:06:20.400 Once you find those top five quer documents whatever it may be then we say okay you found the right document now how do you then find the right information within the document and this is where the work with the docysteine team has gotten really great where we've uh worked with them to build the neatron parse model to s49 00:06:20.400 --> 00:06:36.479 focus specifically on table extraction which is a really complicated technique um if you think about it the number of permutations of tables are quite vast when you think about nested tables merged cells merged columns merged rows whatever it may be and that can get really really complex and really hairy of a s50 00:06:36.639 --> 00:06:42.400 Uh so real quick as I mentioned before right our team is responsible for uh leading a lot of the leaderboards in the retrieval space. s51 00:06:42.400 --> 00:06:46.400 So Vidori V1, V2, V3, MTB, MMT. s52 00:06:46.400 --> 00:06:55.840 Um so our team knows how to build world-class retrieval models given a lot of leadership uh given a lot of leaderboard winnings that we've had in the last year or so and then of course s53 00:06:55.840 --> 00:07:06.000 as I mentioned before we open source everything right so we share the open source model weights the techniques and then we release with those blueprints and skills that agents can use then afterwards s54 00:07:06.000 --> 00:07:14.400 um so to touch a little bit on the actual model that we are working with with docuign was the neatron parse model so when you think VLM you generally think a multi-billion s55 00:07:14.400 --> 00:07:17.599 parameter model it's very heavy It's high latency. s56 00:07:17.599 --> 00:07:21.840 Um, this is a very small tiny C radio VLM. s57 00:07:21.840 --> 00:07:24.800 It's about 850 900 million parameter model. s58 00:07:24.800 --> 00:07:37.120 Uh, designed to kind of be that all-in-one package sort of model where you deploy it and instead of having small, let's say, YOLO X models that do table extraction or page element extraction, whatever it may be. s59 00:07:37.120 --> 00:07:47.840 This is a singleshot model that you can feed a document in and out comes the semantic formatting layouts the text uh the reading order uh the the preserved structure of the table etc. s60 00:07:47.840 --> 00:07:52.560 Um this can be served via the NVIDIA NIM or via the LLM as well too. s61 00:07:52.560 --> 00:07:54.879 Um and so it's a tiny small model that you can use. s62 00:07:54.879 --> 00:07:55.680 It's not a generator. s63 00:07:55.680 --> 00:07:59.039 It's more of an extractor at the end of the day. s64 00:07:59.440 --> 00:08:05.840 Uh so real quick as well too um we always want to make sure that we're building towards benchmarks that matter most to the enterprise space. s65 00:08:05.840 --> 00:08:14.000 So we want to make sure that both on the paro curve of accuracy versus performance, we're make sure that we're going to be releasing a world-class models to the ecosystem too. s66 00:08:14.000 --> 00:08:17.199 So you'll see here generally is just a very standard benchmark of table extraction. s67 00:08:17.199 --> 00:08:30.479 I believe this one was RD table bench and we compare some popular open source models here and then we compare how our neatron parse model does compared to that industry and we continue to kind of strive to improve this as time goes on. s68 00:08:30.720 --> 00:08:32.800 So that I think we believe we have a demo as well. s69 00:08:32.800 --> 00:08:34.399 Yeah, s70 00:08:34.399 --> 00:08:35.440 just press one. s71 00:08:35.440 --> 00:08:35.760 Yeah. s72 00:08:35.760 --> 00:08:36.479 Okay, there we go. s73 00:08:36.479 --> 00:08:47.920 Let me show you how easy it is to turn any agreement into structured usable data with agreement manager, which is a central repository of every agreement an organization has ever signed. s74 00:08:47.920 --> 00:08:49.839 Let's look at this. s75 00:08:49.839 --> 00:08:59.760 So when we look at the agreement manager view here, you know, we have an ability to see the entire list of agreements, but also go and upload a new agreement. s76 00:08:59.760 --> 00:09:03.680 So I'm uploading a new order form into agreement manager. s77 00:09:03.680 --> 00:09:07.839 As you can see, I can select from my computer, import from other places. s78 00:09:07.839 --> 00:09:14.000 The moment I select the agreement, it starts uploading and starts processing with AI. s79 00:09:14.000 --> 00:09:18.480 And just like that, you can see that the jobs engine has processed it. s80 00:09:18.480 --> 00:09:21.440 Let's take a closer look at this agreement. s81 00:09:21.440 --> 00:09:25.040 So when you go into the action, you can go and browse the file. s82 00:09:25.040 --> 00:09:29.440 Within seconds, agreement manager has extracted a rich set of metadata. s83 00:09:29.440 --> 00:09:40.640 Everything from key terms to commercial details are automatically structured, highlighted, and immediately you can jump to that section where the details are found. s84 00:09:40.640 --> 00:09:42.080 Builtin goes deeper. s85 00:09:42.080 --> 00:09:48.880 This is where the Nvidia's model comes in that it's extracted all the structured pricing data around this agreement. s86 00:09:48.880 --> 00:09:54.880 It goes in breaks down these complex t tables into order details. s87 00:09:55.120 --> 00:09:58.800 And as you can see, we can break it down. s88 00:09:58.800 --> 00:10:00.880 We can download all of this data. s89 00:10:00.880 --> 00:10:09.680 This is powered by the advanced parsing leveraging Nvidia's Neotron model, turning every even dense tables into something that is instantly usable. s90 00:10:09.680 --> 00:10:12.000 And of course, you can take this data with you. s91 00:10:12.000 --> 00:10:19.200 You can see when we've downloaded into CSV how we've structured all of it for your finance team, procurement team, even further analysis. s92 00:10:19.200 --> 00:10:22.160 All of it is also available through API. s93 00:10:22.160 --> 00:10:30.480 And that's how agreement manager has transformed agreements into actionable insights in seconds leveraging Nvidia. s94 00:10:30.560 --> 00:10:35.760 So I think you know what you saw there from um a demo perspective we've tried to shortened it. s95 00:10:35.760 --> 00:10:47.040 It's like we have a whole repository what you see a list we get customers which has thousand agreements to all the way millions of agreements within but the big piece is how do we understand and get that data s96 00:10:47.040 --> 00:10:59.120 that makes it very valuable to an end business user right a legal person a procurement person a salesperson who's doing a lot of the deals or even a leader right like a business unit the CTO goes and asks what did we do s97 00:10:59.120 --> 00:11:09.760 this is how we have making each of the things a lot more structured so we have our own proprietary agreement data model which we are structurizing each agreement but also at a whole organization s98 00:11:09.760 --> 00:11:19.600 level and leveraging a lot of the NVIDIA things we've been able to do a really good job especially with all of those tables like pricing SLAs's and then make that available s99 00:11:19.600 --> 00:11:30.320 and then we also have a like robust kind of search that is um on top of it so when we think about what have we learned right when you think from a neotron plus docyign s100 00:11:30.320 --> 00:11:36.160 we one of the biggest things for us we definitely have done lot of different models mod for different purposes. s101 00:11:36.160 --> 00:11:47.440 So a purpose-built model for the job you're trying to do is a big big part of how we've been thinking about and that's kind of where we've been able to accelerate bring things to market much faster. s102 00:11:47.440 --> 00:11:50.079 The second big piece around like the model efficiency. s103 00:11:50.079 --> 00:11:58.880 So for you know as Sean was talking about the number of parameters yes context and stuff matters in the you know in a different environment for different things. s104 00:11:58.880 --> 00:12:09.920 For us the lower kind of context basically also meant lower latency lower cost to deliver the scale that we are talking about last around the faster extraction. s105 00:12:09.920 --> 00:12:21.600 So um we we ran this against a lot of the other open source models when we think about how many tables can it extract per second Neotron was 20x faster s106 00:12:21.600 --> 00:12:28.560 which helps us when we're talking about the millions and billions of scale that we're kind of serving for all of our customers. s107 00:12:28.560 --> 00:12:41.279 So a lot of it is like having that smaller purpose-built things is the way for an enterprise as an organization to go and leverage and then serve that from an enduser perspective. s108 00:12:41.279 --> 00:12:42.880 Um and then what's next? s109 00:12:42.880 --> 00:12:45.040 So I'll let Sean talk through those. s110 00:12:45.040 --> 00:12:45.279 Yeah. s111 00:12:45.279 --> 00:12:48.639 So working with the docuign team uh has been awesome so far. s112 00:12:48.639 --> 00:12:51.839 Uh and we're going to continue to deepen that partnership as well over the next few months. s113 00:12:51.839 --> 00:12:59.920 So uh with them we started with the how do I extract as much possible information from a page and now we'll scale to how do I now find that page to begin with. s114 00:12:59.920 --> 00:13:08.560 Um so we'll start a little bit with the neatron retriever uh effort and then of course we'll talk a little bit about the NVIDIA agent toolkit with them over the next few months s115 00:13:08.560 --> 00:13:12.320 um and then actually start scaling into into more production scale agents then. s116 00:13:12.320 --> 00:13:13.120 Perfect. s117 00:13:13.120 --> 00:13:14.639 I think that's what we had. s118 00:13:14.639 --> 00:13:17.440 We have time for a couple questions. s119 00:13:17.440 --> 00:13:20.079 I went in the room. s120 00:13:22.720 --> 00:13:23.680 Okay. s121 00:13:23.680 --> 00:13:26.000 Someone there. s122 00:13:44.880 --> 00:13:55.760 So just to recap for everybody if you didn't hear it was the question is right like OCR is always a thorn in the whole process so are we thinking about letting the go of that and starting from agentic from the get-go s123 00:13:55.760 --> 00:14:07.519 I can talk from my perspective but so I think for us right like there are different use cases at different points in time many times if you are reactive you have a question and you're coming some of that can can s124 00:14:07.519 --> 00:14:09.839 work dynamically at a smaller scale. s125 00:14:09.839 --> 00:14:17.199 The question is the latency when I am quering at that scale of thousands I do need to have pre-processed have identified. s126 00:14:17.199 --> 00:14:18.079 So that's one. s127 00:14:18.079 --> 00:14:25.360 I think the second big part of the use case for us a lot of times businesses want to use this data to do a lot of downstream work. s128 00:14:25.360 --> 00:14:27.199 So an example is a procurement team. s129 00:14:27.199 --> 00:14:28.399 This is my pricing table. s130 00:14:28.399 --> 00:14:37.279 I want to put it into Koopa to make sure when I'm paying that works at that time there like you know the agent is kind of helping but I can't do that on a one document by document. s131 00:14:37.279 --> 00:14:39.519 That said, there is ways that we are compressing. s132 00:14:39.519 --> 00:14:47.199 That's kind of why Neotron worked for us is like how do you do it from a layout understanding just for that purpose but I would like let you add. s133 00:14:47.199 --> 00:14:49.680 Yeah, I think it depends on the use case a little bit. s134 00:14:49.680 --> 00:14:56.800 Um I think for this specific instance, right, you have pabytes of documents that you want to be queriable at some point, right? s135 00:14:56.800 --> 00:15:02.320 So you are heavy on the compute at the upfront side with all the OCR so you don't have to worry about it later on, right? s136 00:15:02.320 --> 00:15:08.240 I think there's some instances where people may upload a contract to begin with for Q&A and that's a very high that's a very low latency use case, right? s137 00:15:08.240 --> 00:15:22.240 So you have a high throughput versus low latency use case and in that scenario your different batch sizes, your concurrencies, your different techniques on how you process the document will be different and where you spend that compute in that cycle will be changing between the different use cases. s138 00:15:22.399 --> 00:15:25.600 Okay, one more there. s139 00:15:35.600 --> 00:15:42.079 We we do a lot of like more of what I call hybrid approach and a purpose-built for like the needs and the use cases. s140 00:15:42.079 --> 00:15:52.880 So from a table piece it does kind of you know do the whole layout along with extracting we still do OCR from a lot of other fields and metadata and a closet like all of the text kind of thing. s141 00:15:52.880 --> 00:15:58.320 So that I think we had a architecture where we have our pipeline going through two different routes for that. s142 00:15:58.320 --> 00:16:05.199 Um as a follow we have a blog out there how are we really solving this at scale across and if you look at that there's a lot of different s143 00:16:05.199 --> 00:16:08.079 peacemail modules and stuff together. s144 00:16:08.079 --> 00:16:10.560 Yeah one moreization s145 00:16:14.160 --> 00:16:22.079 so we this model is currently on FP16 but there are paths towards going on to FP8 and VFP4 in the next few months as well too. s146 00:16:22.079 --> 00:16:23.360 What about center? s147 00:16:23.360 --> 00:16:26.320 Are you using the 16 or s148 00:16:26.320 --> 00:16:34.720 we do use that and then we're also kind of using some of the older ones and that's the journey as a partnership is to kind of go tweak as you get more of the customers. s149 00:16:34.720 --> 00:16:34.959 Yeah. s150 00:16:34.959 --> 00:16:42.000 So for this there are many techniques on how to improve the performance side right so quantization right so we're trying to move everyone to blackwell right so that's why NVFP4 s151 00:16:42.000 --> 00:16:58.880 is the big thing now um as well as multi-token generation for this it's a VLM architecture right so your encoder decoder techniques can definitely be further optimized so not right now this model just generates one token at a time you can do multi token generation of course too so there's plenty of performance things right now we're focusing s152 00:16:58.880 --> 00:17:08.079 on the accuracy side like are we adding value to the system and then from there we'll then push out that paro curve on the performance Are you guys using s153 00:17:13.839 --> 00:17:21.760 I believe they just deploy via directly recall when you do that. s154 00:17:29.520 --> 00:17:31.360 Oh, this is just an extraction. s155 00:17:31.360 --> 00:17:32.640 This is now a retrieval. s156 00:17:32.640 --> 00:17:33.120 Yeah. s157 00:17:33.120 --> 00:17:33.760 Yeah. s158 00:17:33.760 --> 00:17:35.760 Yeah, you're asking question. s159 00:17:35.760 --> 00:17:41.280 It's coming from that agreement data that we've kind of extracted and stored. s160 00:17:41.840 --> 00:17:48.880 So yeah, maybe I can chat with you offline and how like we our architecture kind of works fully as well. s161 00:17:49.039 --> 00:17:51.120 No, we're almost coming up on time there. s162 00:17:51.120 --> 00:17:53.919 Um, but I think that's kind of all we have. s163 00:17:53.919 --> 00:17:57.600 I'm happy to hang around uh in the back with more questions. s164 00:17:57.600 --> 00:18:01.039 Um, and good luck with a lot of your uh challenges with AI. s165 00:18:01.039 --> 00:18:01.760 So, thank you. s166 00:18:01.760 --> 00:18:03.382 Thank you. s167 00:18:03.382 --> 00:18:05.382 [applause] s168 00:18:18.447 --> 00:18:20.447 [music]