From Raw Documents to AI-Ready Data — Leo Platzer & Jeff Koss

AI Engineer · 21 min · 246 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:00All
00:01[music]
00:13right.
00:15Hey guys, let's get going.
00:16Can everybody hear me?
00:18All good.
00:19Perfect.
00:20All right.
00:20My name is Jeff CS.
00:21I'm a staff customer engineer with DZ Labs.
00:25Um, as you can see from the slide, it says brought to you by Calibbra.
00:29Calibbra acquired DZ Labs last summer.
00:32Uh Calibbra is more of a data governance tool for structured and AI um agents and models.
00:37[snorts]
00:38So the gap that Calibbra had was we didn't have unstructured and that's really where uh DZ shines.
00:44So today uh with me is Leo Platzer.
00:47Yeah.
00:48Hi everyone.
00:48Uh I'm Leo.
00:49I was previously the CTO at DZ Labs which as Jeff has already mentioned was acquired last year by Calibbra.
00:56And you can see we're both twins as well.
00:58We have our hats that we're giving away at our booth as well.
01:01So come by and get them.
01:03First come, first serve.
01:04So today we're talking about um how we can go from a stalled PC to production with DZ data curation.
01:11So really focus on unstructured data.
01:15So our demo scenario I typically talk about is a manufacturing uh company, North River Manufacturing.
01:21You can see that you know there's 40 40,000 employees.
01:25They operate in 20 different countries.
01:27Uh really what they're trying to do is they have millions of documents in SharePoint.
01:31They've accumulated them over years and years and they need to figure out which ones are relevant for their chatbot.
01:37They're invested heavily in AI to improve operational efficiency.
01:41And their big objective this year is they want to build a legal ops AI chatbot.
01:48Now what they want to do is they want to help the legal and procurement teams answer questions around supplier contracts, partnership agreements, and it's going to be real time.
01:58So some of the questions they're trying to answer or what are the termination clauses for a specific supplier?
02:04Um which agreements expire this quarter, which agreements expire next quarter, and what partnerships have u exclusivity terms.
02:14So some of the folks in the teams that we've been working with um is Sarah.
02:19She's a head of data and AI.
02:20So she's responsible for the accuracy.
02:23Um also there's Jim.
02:25He's the AI engineer.
02:26He just needs to know which files do I need?
02:29I have tons and tons thousands and thousands of files in SharePoint.
02:33I need to find just the relevant ones with no sensitivity.
02:37Priya is the head of knowledge management.
02:39She wants to make sure that no sensitivity is leaked.
02:43um and make sure that if there if there is specific data that's uh classified that it gets redacted as well.
02:51Then there's Michael.
02:52He's a director of legal ops.
02:53He needs a chatbot to deliver accurate responses to his team.
02:57So he's from the business.
02:59And then finally there's Jackson.
03:01So Jackson's enterprise transformation uh lead and this is his flagship use case but also he has 15 other use cases that he needs to to deal with right so he wants to have a framework a repeatable
03:17successful process he doesn't just want to do it for one time he wants to do it for multiple different use cases so what they're able to do so far they were able to take 40 around 40 different files
03:34pre-curated files and build a small little PC small little mini pilot where they can do uh parsing chunking OCR embedding and then put it in a vector database and a knowledge graph and they got pretty good responses
03:50right seems pretty easy you can do that with 40 files but when they try to do it with many many different files with 80,000 plus files.
04:02Right now a sudden it's a little bit more challenging.
04:06And so some of the challenges they had were I can't discover the files that I need.
04:11I don't know where the sensitive data is in my on SharePoint.
04:16I can't trust the quality of the data.
04:18So my background um I used to work at both Informatica and data bricks.
04:25So for me data quality is typically the six different dimensions of data quality.
04:29Completeness, conformity, accuracy.
04:32But when you deal with documents, it's a little bit different.
04:35It's I want to see is there duplicate information, conflicting information, right?
04:40Leo's going to talk about he has a really good slide about showing why it's so important and also the freshness of the data and also everybody's talking about context, right?
04:51So what we want to do is when we tag the metadata, we want to use that in our chatbot in our vector database to make it more relevant, have better accuracy in the chatbot.
05:04So you can see some of the challenges that they had, right?
05:08They're getting incorrect answers.
05:11There's sensitive information that's being leaked out in the chat bots.
05:15It takes a ton of time to go to production because they can't find the files.
05:18is spending too much time looking for the files instead of writing code and doing things that they they really like to do, right?
05:25It's difficult to maintain.
05:26I'm working with another telecommunications company and they have six to seven different SME and they're trying to curate 20,000 HTML files and they can't keep up, right?
05:38So, it's very brittle, very difficult.
05:40And what we do is we provide the ability to automate this entire process.
05:46So over here now you can see they want to they want to try to access and leverage over 80,000 different files.
05:55So they went from 40 files to over 80,000 files.
05:59And really what they want to do is they want to find which files are relevant for their chatbot, which have high quality, which ones are up to date with knowledge.
06:07They don't want to use a a version where the contract is is old and expired, right?
06:12and they want to enrich their chatbot with the metadata that they've tagged and of course they don't want any sensitive data and so what we do over here is we provide the ability to do that right we can access the data
06:29um over here we can understand what the data is we can figure out which files are relevant do the OCR the chunking and things like that so let me show you a little a brief demo So
06:42over here you see in SharePoint much like what North River has they have a whole SharePoint site full of different you know there's call transcripts there's PDFs there's excel there's word documents
06:54and I they don't know which ones are relevant which ones are focused around legal now over here when we look at DZ we've already opened up the application right here and I can see you know I've already ingested 339
07:07files and now I can see both the uh page page level or chunk level as well as a file level.
07:14I can see all the information right here.
07:18And then over here, this is what we call a texonomy.
07:21Taxonomy is a grouping of tags or a tagging tree.
07:24So I've ingested the 339 files.
07:26And now I can either manually create a userdefined tag.
07:30I can leverage tags that are reusable for my tag library or I can leverage AI to autosuggest what tags I should even have.
07:38Right?
07:38I can leverage AI to say you these are the tags are relevant.
07:42So here I just type in my tag name data document categorization give it a prompt and this prompt is what we're going to send to the LLM to determine
07:52is this file how should we tag this file now if I know what I'm looking for I can put in the classification values it's optional and again if I can leverage AI to autosuggest
08:03all these tags for me now also the other thing to point out if I have my uh my taxonomy here, my valid values um already.
08:14I can upload them through a CSV.
08:16And so once once we do this now, I can see I started to build this this tagging tree.
08:22Now the other thing I can do is I can have conditions on my tagging tree.
08:26So remember I said I only want legal and contracts.
08:29That's what they're looking for.
08:30Now what I can do is I can have have um subtagging.
08:35So only legal and contract files.
08:37I want to pull out more data around the contract type.
08:41Is it a supplier contract?
08:42Is it part of an NDA?
08:44Right?
08:45Is it materials and and time contract?
08:53And now what we can do when we add this, you can see we have different branches in our tagging tree.
09:01And I could have um unconditional tags as well.
09:05I can have any number of different tags.
09:07Now, what we're doing is we're going to generate the metadata.
09:09So, I choose what files I want.
09:12I can choose what tags I want.
09:14I can say generate metadata.
09:15And we'll run this.
09:16And you'll see the results over here.
09:22So now I can see the document category.
09:24I can hover my mouse.
09:26There's engineering documents.
09:27There's my contracts and legal, right?
09:29I have 68 different files.
09:32And if I click on any of those, it automatically filters for me.
09:37Now I can also see the evidence.
09:40Why did AI tag this?
09:41Where in the document did it find evidence that it's a legal contract?
09:47Right?
09:48And I can also do thumbs up and thumbs down.
09:49So there's fshot learning as well.
09:51So I can give it an example if I want to.
09:54Now you can notice I also have filters.
09:57So I have two different filters.
09:58I quickly went from 339 files down to 20 files.
10:04So, we also have the ability to scan for sensitivity as well.
10:08So, whether it's PII, PHI, um could be anything.
10:14I was in uh in London at a Gartner conference uh a couple months ago and there was some folks from Finland that came up and they wanted to know, you know, can we find Finnish national ID?
10:24I don't speak Finnish.
10:26So, I put it in there.
10:27I did this demo and not only can we do AI based, we can do patternbased with context and it ended up figuring out the the reax um syntax for me, but it also added
10:38some Finnish words which I thought were probably associated with Finnish national ID.
10:43So, it's going to be less false positives as well.
10:52All right, let me keep going here.
10:55So the other thing we can do is we can do quality.
10:58So again for us we have a dashboard in here for quality where we can go through and see what are the different groupings for um for duplication for conflicting information.
11:09Um and then we'll we'll leverage AI to auto suggest which ones to keep.
11:15Now the other thing we can do too is again for freshness we can say I only want files within x number of days or within a specific date range
11:23and then the last thing we can do for data quality is you know here's the the pieces of metadata that I absolutely have to have and then what we'll do is we'll figure out you know of the 339
11:36files maybe you really need 200 files and then from there you can create a data slice and then we can also give it context as uh over here.
11:47So I'll go through one more video and we also have a live demo for you that Leo is going to be doing in a second.
11:54So in this case now what I can do is I can I can leverage a a data slice.
11:58So my data slice could be filtered based on the specific criteria that I want.
12:04So in this case I have 14 files.
12:06These are only the files that are relevant for legal.
12:10And now what I'm doing is I'm copying my use case description.
12:13What I can do is I can say here's what I'm trying to build and I can either do a flat uh tonomy or multiple layer levels in my tagging tree
12:23and then over there when I run it it's going to autosuggest these tags for me.
12:26There's human in the loop so I can choose which ones I like, which ones I want, which ones are relevant.
12:31But again, it's all done for me based on what I'm trying to do.
12:36We gave it context and based on the content of the files themselves.
12:42And then over here it even builds out the classification values for me.
12:50And then if we want when we run when we generate the metadata we're the job now we can see over here's all the sensitive information right here's all the different information that we have over here.
13:04And then again I can see the evidence at the file level and the chunk level.
13:08I can see the confidence score.
13:10So coming back from from AI what the model return I don't have to guess you know it's 100% confident 95% confidence.
13:17Here's why.
13:18So you can have very a lot of confidence in the the um AI ready data product that you guys are going to be building.
13:30All right.
13:33And now finally what we can do uh for delivery and maintenance.
13:37So you know one other thing that that's really important is with our data slice technology we have workflows where you can put on a set schedule so it's auto refreshed.
13:46So we want to make sure that your chatbot never gets stale.
13:49So if there's new records or sorry new files that get added to SharePoint or deleted, we'll automatically pick those up and add those on a set schedule to my
13:59um uh to my data slice or my AI ready data product and then our SDK will pick that up in the chatbot and then when we load the Victor database, you'll always have the most freshest
14:10uh metadata.
14:14So what's it mean for North River?
14:16So basically they were able to go from four months of data prep work to down a days.
14:21They were able to increase their data utilization for AI.
14:25They increase their chatbot accuracy because they not only were able to figure out which files to use but also the metadata from the files.
14:33They were able to inject that and leverage that through our SDK in their chatbot.
14:39Right?
14:39And then the other thing they did was they reduced their compliance risk.
14:42they knew where all the sensitive data is and they made sure that they were not part of they were all filtered out in their u data stores.
14:52Okay, so I'm going to turn it over to my brother from another mother, Leo, and he's going to talk about um context.
15:00Yeah.
15:00Hi everyone.
15:01Um I just wanted to basically like give this so what right technically okay you dduplicate your data, you reduce your search space, right?
15:11you clean it up, you have it fresh.
15:13What does it actually mean?
15:14Right?
15:15So, if we were looking at the vector space, right, when you ask a question, it comes in and you have duplicated data with the same information or potentially even
15:25a different version of the document with conflicting information now, right?
15:29Think of your Slack messages.
15:31Today, a person says this, tomorrow a person says that, right?
15:34The truth changes over time.
15:37If it ch if it stays however in your knowledge base in your search space your top K will always be full of pretty much irrelevant information right so in DZ for example
15:49something which is considered duplicative is either another piece of context like your database has changed and I can actually use it or it is simply redundant and we will remove it from the knowledge base.
16:04The second piece is like same as same reason as before.
16:08If your information is old, right?
16:11If you suddenly are working with data from before the 2000s, if you're working with information which was only relevant last week is not relevant this week, in the future some of the use case when agents actually get into production for let's call it the most high stakes use cases
16:26that has detrimental effects.
16:30And we have done um a lot of evaluation around what is the effect of making sure that your unstructured data quality is maintained, is consistent and is always handled
16:44deterministically in the same way, right?
16:48And what we're essentially seeing is that if your raw corpus, okay, let's imagine you have a thousand files and 30% of your files are stale or duplicative, it can happen that up to 80% of your actual context
17:06your agent uses to answer a question is filled up with stale information.
17:12which means like it actually uses 80% of completely redundant useless information and knowledge and this has to be managed.
17:23Um what we have seen is that by running on the exact same tasks.
17:27So this is a rack chatbot.
17:29It's not um running an actual like let's say cloud code task or codeex on a coding task.
17:35It was simply like multihop rag evaluation.
17:40And what we saw is that if we look at recall one like I mean we pretty much double uh the recall um and overall improve our accuracy.
17:50So like task completion by between 10 and 15%.
17:55And why is this so significant?
17:57This is so significant because this has nothing to do with the use case you're working with.
18:02This is true independent of what you're actually using the data for.
18:08Right?
18:08Because if you actually know what you want to use your data for, you can apply relevancy and all not other data quality dimensions, right?
18:15But duplication and freshness is true no matter what.
18:20And yesterday, right, there is a lot of context topic, a lot of context flags around here.
18:27Um, I wanted to give a very very quick demo on how you would use the DZ SDK to prepare a context repository.
18:36And I'm going to run this live.
18:39So what this is doing is essentially it's going to fetch all of the metadata we have about a given SharePoint site and we will then contextualize each folder within that SharePoint site as well as create an index
18:53over all folders so that your claw code for example or your codeex does not even have to look at the individual files to figure out what to look for but simply looks at the context
19:04MD file.
19:06So what you're seeing here is essentially really by using DZY you can curate context files where you are directly aware what exactly is within each folder right what is its purpose
19:23right when do you want to use it so for example this is one to one the mirror of what I actually have in SharePoint but now instead of actually looking at these 23 files
19:35figure out whether I even should look at this folder.
19:38I know exactly what document types I have, what it is about, what are the key topics, when to use this folder, and what are example questions.
19:48So this is how this is all the kinds of use cases today, right?
19:53How you can use the easy for use cases like your rack chatbot or use the easy to actually improve the performance and reduce, for example, the token cost for your agents.
20:03Um, but yeah, basically I'll move over to Jeff again to summarize this whole presentation.
20:12Thanks, Leo.
20:13All right, so guys, um, we're almost out of time.
20:17Our booth is way in the back by stage two.
20:21Please come by, bring any different use cases that you guys have.
20:25Um, we're happy to have discussions, do live demos, um, talk about what your challenges are around unstructured documents.
20:33um over here.
20:34Um before we leave, just want to let you guys know we have oneweek PC's.
20:40So, we're happy to prove our technology.
20:42Uh again, you know, you guys bring your your business problem.
20:45Um you're trying to solve with AI, bring some sample data, a bunch of different files, and then what we can do is we can get generate metadata for each different files.
20:55Um we could get a full sensitivity report on your on your data, data quality report as well.
21:01and then we'll show you how we can build a full tonomy based on your specific use cases, your specific files.
21:08So with that, everybody, appreciate you guys spending time with us.
21:13And lastly, we only have one more case of these hats, so come on by and grab them.
21:17So we'll be out in the back.
21:18Thanks, guys.
21:19Appreciate you.