Parsing the Epstein Files for JMail — Palak Agarwal & Omar Alhait, Reducto

AI Engineer · 29 min · 324 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:12Hi everyone, my name is Ple.
00:14I lead developer redu relations at Reductto.
00:17Um, and before we dive into Jmail, I'd love to give you a little background on what we do.
00:22Uh we're an agent document platform which basically means we help teams turn messy unstructured documents into data that AI can actually use accurately.
00:32Um to give you some context in the last three years we've raised 108 million from investors like ACCNC benchmark and more and processed over two billion documents across customers like Harvey Scale and a top five global hedge fund.
00:47Um, and operating at that scale means that we've basically dealt with every document you can possibly think of.
00:53Um, and we've iterated reductive product to handle anything.
00:58Um, now getting to AI to work on a clean PDF with a frontier model is not that hard, but the truth is that most of enterprise documents are unstructured.
01:09That means like scan documents, rotated documents, charts.
01:13Um, and that's when things get really complex.
01:16Reductto is the layer that sits right before your model.
01:19Um we handle complex layouts but also um as well as post-processing so you can focus on building your actual product.
01:27Um the reason this works is that we take a rasterizationbased approach that means we look at your document like a human would look at the document instead of the underlying embedding.
01:37Um everything is powered by our agentic OCR framework where rather than making a single pass and committing to an answer the model flags things that it's uncertain about corrects them and then moves on to the next step.
01:50Uh we'll give you access to our studio.
01:52Um we've made it self-s served just for this session.
01:55Uh but in total we have five core capabilities.
01:58Parse, split, extract, edit, and classify.
02:02Um now a quick show of hands.
02:04Who here has heard of Jmail or looked through Jmail before?
02:09Okay.
02:09Wow, that's a that's a good chunk of people.
02:12Um, but I'll I'll give a quick rundown in case you haven't.
02:15So, Jmail was originally built by Riley Walls and Luke in five hours last November.
02:21That's when the Epstein files were released.
02:22Uh, the concept is basically Gmail forked and you can log into the inbox like your Epstein and then that grew into a full suite.
02:30So you can think of like J drive, JFlight, J, Amazon, J photos and so much more.
02:35Um by February there were over 450 million visits and 18 million users that had access Jmail.
02:43And we're really proud to share that Reduct was behind parsing all of the documents.
02:47And that's not a small number.
02:48It's over three million pages of redacted PDFs, scanned documents, and messy government files as well.
02:55And to give you an idea, like this ranged from anything like really complex forms and tables to like messy handwriting of like so many different people.
03:04Um, now I'm going to hand it over to Omar, who's one of our founding engineers and worked very closely with the Jail team back in November.
03:11Um, and he'll give you a technical rundown and more.
03:14Alrighty, who's ready to talk about some documents.
03:18Yippee.
03:20All right.
03:20So, um, yeah.
03:23So, uh, walking us back to, you know, a very fateful night in November, right?
03:27Um, I was talking with the founding team and we were trying to figure out, you know, what would be a great way to use Reduct with this kind of
03:35transformational cultural event that was, you know, going to be unveiled to the public, right?
03:43Um it just so happened that um you know we're very close to Luke uh who is a founder at Kino and is working downstairs and we were all just kind of talking about you know what would be cool to build what would be funny.
03:54Haha.
03:56Um and then uh it kind of snowballed into this idea of like what if you could um you know sign into your email as if you were Jeffrey Epstein
04:06and view um you know your email as if you were him.
04:10And so um you know it was a really really cool concept as a product but there was one really hard problem to solve that you know was basically a barrier between us and the productized
04:22solution.
04:23um and you're looking at part of it, right?
04:26So, it's really really like I I would urge you if you haven't done this before to send this like uh you know uh cover sheet to Chad GPT and try to get some results out of it that and and just see how accurate it is, right?
04:41It's a very very complex problem where you know layouts are um pretty hard to understand.
04:47The reading order of this is also somewhat subjective, right?
04:51Um, and so we needed something to basically solve this bottleneck so that we could work on all of these cool uh little, you know, products downstream.
05:02And so this is where Redto came in.
05:04So I'm actually going to show you guys um what we had used for a couple of our products, right?
05:10So Oh yes, let me do this real quick.
05:22Alrighty.
05:24Whoops.
05:25Gonna bring this over.
05:27All righty.
05:28So, here you see a cool product, right?
05:31JF flights.
05:32Basically, all of Jeffrey Abstein and his affiliates flight history, right?
05:37This guy was flying people out.
05:38He was flying diplomats out.
05:39He himself was flying many places to and from his island, right?
05:44Um what was basically the barrier between us and finding out finding all these flights were these pretty shoddy handwritten documents, right?
05:53Um again, if you were, you know, if you thought that cover sheet was a little bit rough, think about handwriting, right?
05:59So we can see here a pipeline that I had actually set up way back when uh when we were trying to build a good config to basically make our parsing really really good, right?
06:14And so in this case uh we had for the for the flights use case, you know, an idea had flown out that like what if we um you know spin this off as a completely separate product where you could just see all of his flights.
06:26Um and so what that meant is taking all of his flight documentation and then applying a separate you know extraction config for it.
06:34And so what you'll see here is um kind of a little tour as to how Reduct works, right?
06:40So Reduct uh within the document extraction use case first parses the document, right?
06:45And so this kind of shows you, you know, you could stick any document that you would like here and you could just click run, right?
06:51Configure any of our bells and whistles and see real-time results, right?
06:56So this was sort of the canvas for myself that I was playing around with to kind of understand before I start, you know, scaling this up to a million different documents and run up a quick check on Reduct's,
07:06you know, on Reduct's uh um you know, uh receipts, let me make sure that our um extraction accuracy is what I would expect it to be.
07:17And so here is kind of what we ended up with.
07:20Um you could see here for a couple of these different documents, you know, some redactions that we had to deal with.
07:26Um Woody Allen, I know that guy.
07:28Um but yeah, just like a lot of different um you know, use cases that we had to handle through one extraction schema um for the flights use case, right?
07:39And so we parsed all the documents uh very quickly.
07:43And actually a fun story about this um is Alvin in the room?
07:48Alvin is in the room.
07:49So, um, back when the, uh, the, uh, first, uh, drop came out, I was paging Alvin maybe like, you know, a couple times a night, uh, he was, he just happened to be on call,
08:02uh, the night that I was parsing all these documents.
08:04And we were trying to parse them very, very quickly.
08:06And so, we put a, you know, a decent amount of stress on our infra.
08:11And so, Alvin was sending me a lot of fun messages late at night like, "Hey, bro, can you slow down?"
08:16Um, but yeah, we got through them.
08:17We got through them in time.
08:18And then um once we had this kind of corpus of documents that we were able to um you know um kind of take as a ground truth, we then spun off a a billion different products, right?
08:31And so Paula kind of touched on this a little bit, but I'm going to go back to Gmail.
08:40So, you know, we we spun up any anything from Jazm to Jlights to Gemini, right?
08:46Um yeah.
08:48So, and I and also JCAL.
08:49This was actually a product that I um worked on myself.
08:54So, some fun stuff there.
08:56Um cool.
08:57And so, I guess you know if you could kind of take uh anything out of this, I do want to say actually um yeah, so if you could take anything out of this, right, like the simplicity from this is really like
09:13you know getting a schema, right?
09:15looking at these results, making sure that the results look like what I would expect, and then just deploying.
09:21Right?
09:21So, if I was to deploy this code snippet, I'll just redeploy here for our use case,
09:27[snorts]
09:28and I can get either a direct API call or just a pipeline code snippet that I can invoke very quickly, right?
09:35Um so the name of name of the game here is like stick your documents in this platform and then immediately start um you know visualizing your data and you could immediately also productionize it uh afterwards.
09:47So I'm going to stop here real quick.
09:49Um do we want does anybody have any questions thus far about um Jmail or Reduct in general before we can kind of start uh working on a little interactive exercise?
10:02Yeah.
10:05Yeah,
10:11great question.
10:12So, we can actually come up with this for you, right?
10:14Um, you know, my uh my fun analogy here is that like your extract schema is sort of like the paintbrush on your reductto canvas, right?
10:22So, it definitely drives everything that you um you know, you get out of the schema.
10:28So, for example, right, I could create a new pipeline here, right?
10:32and create a new extract pipeline and just call this I don't know let me make a resume schema because I want to parse some résumés right
10:45oh whoops sorry about that actually I'm going to do this real quick so anyway I I'll do this again because it's very simple so I'll create a new extract pipeline here by the way we could do a lot of things with documents par split extract, edit, even classify.
11:04Um, and so what I'm going to do here is create a resume schema.
11:12Oops.
11:15Uh, my Wi-Fi is not being helpful with me today.
11:19There we go.
11:21All righty.
11:23So, I got a fun schema here.
11:24And now it's up to me to basically decide whether or not this schema is like satisfactory or not.
11:29Right.
11:30it it's really given it's going to give me exactly what I'm asking for and you know once I plug a document in here I might realize like ah this is actually not what I want right so
11:40I have a bunch of handy dandy resumes I'm going to stick them in here upload and then just run this so yeah um I hope that answered your question
11:58any other questions while We wait for this.
12:09How do you validate?
12:15You're talking about the loading state.
12:17Uh that's just like a fun loading state at the end of the day, right?
12:19Like our product is like a really, you know, it's a suite of different models uh and different VLMs that we deploy, right?
12:26Um I'm I'm
12:27sorry, it's not the I was following what you were saying.
12:34Yeah.
12:38Oh, that's a great question.
12:39So, if I'm understanding correctly right, what you're saying here?
12:44Yeah.
12:44Yeah.
12:44So, this specific um this specific like schema generation that I had just used is really out of the box and it's just kind of like a a quick LLM call that describe like builds a JSON schema exactly for what I asked for.
12:57However, we do have an enhanced schema generation, right?
13:01And so what this will do is let's say I've parsed my documents, it will then take the output of like what's in the documents and tailor a schema description directly for them.
13:11So like if you're looking for example for, you know, something just really quick to just get up on the screen and just see some results, I would recommend regular, you know, uh schema generation.
13:21But if you are a lot more interested in like nuance, uh I definitely suggest parsing the documents first and then kind of passing them in.
13:29All you would have to do is just click enhance schema generation.
13:31And what we will be able to do is um you know run the documents first, try to get an understanding of what's in the in the documents so that we can tailor these schema descriptions
13:41directly for that.
13:43Make sense?
13:44Cool.
13:45Cool.
13:45Cool.
13:45Cool.
13:48you had some reduction.
13:51Yeah.
13:54[clears throat]
13:54Correct.
13:55Yes.
13:56So I mean yeah like at the end of the like this is a great question.
13:59Right.
13:59So reductto actually parses PDF metadata as well right.
14:03So some like sometimes uh and this is honestly testament to like uh who was doing the redaction, right?
14:10Sometimes the redaction was actually not that great, right?
14:13And so what that ended up being or the responsibility ended up being on us as the JML team to then have to do a bunch of redaction, right?
14:21Because you can't just stick a box on a PDF and then expect it to actually be fully redacted, right?
14:26Um yeah, seems like there was some uh a lack of understanding when whoever who whoever was cooking up that redaction.
14:37um I will say it's like for the files that came out and myself looking through them that's not the case right some files were actually redacted correctly um but there was some files where we were just able to see who was like you know who the redaction was behind and
14:53um so it became a really really serious you know situation right because we're dealing with you know people's data that like um you our data is redacted for a reason, right?
15:04And uh you want to really make sure that you're not the one that like ends up getting sued at the end of the day because of this.
15:10Um and so yeah, there was a you know a a heavy kind of level of responsibility on our shoulders to make sure that whatever was intended to be redacted was actually redacted.
15:20Our our our tool was working a little too well.
15:24Cool.
15:25Yeah.
15:25at the front and then I'll after you
15:28questions or thousands of different things.
15:38Yeah.
15:38Yeah.
15:39Yeah.
15:39Uh it's a great question.
15:40Um I'll give you some like you know an example, right?
15:45There was there was a schema that I was looking at maybe on Friday uh that had like 156 fields, right?
15:51um and all of these different fields, you know, need like they were requesting something completely different, right?
15:57Uh and so I urge you to kind of put our schema to the test.
16:02I think like our um the like nesting uh limits to our schema or maybe like four or five layers.
16:10Uh but in if you want to, you know, ask for everything in the world, you can still technically be able to fit it in in a schema.
16:19Do you have any capabil resolution
16:23entity resolution.
16:24Can you give me an example?
16:30So this is a good question.
16:32Um there is a lot of post proc postprocessing that we end up doing, right?
16:36Let's say like on the extraction side.
16:38Um I'm personally a little bit farther abstracted away from like the ML side of things.
16:43Uh but I can tell you that we are doing a decent amount of post-processing to basically link um you know entities together.
16:49For example, we have like a citation model, right?
16:52Where we know we have, you know, one model that could be responsible for extraction, right?
16:57But then there might be another model that's also responsible for once this data is extracted, where is each piece kind of extracted from, right?
17:05Uh Nvidia actually did a cool um paper on this uh that they that I just found out about like yesterday.
17:11I think it's called like find everything or something like that.
17:14um where they try to address the bottleneck of like you know citation uh determination and like the speed at which you could find a citation for something that's in a document.
17:25[snorts]
17:25If you're interested about these things, we are hiring and so uh come speak to us after.
17:32Sorry.
17:33Questions.
17:41Yeah, absolutely.
17:43So, let me try to see if one of the résumés I uploaded has an image in it.
17:50So, let's see here.
17:51Right.
17:51So, um we have some figure models, right?
17:54that are basically determining.
17:56Let me actually also rerun this real quick so that it can return figures.
18:18Ah, that's a great question.
18:20Um, Alvin, did you hear that?
18:22No, no. So, I I'll explain.
18:24The reason why um I I asked Alvin is because Alvin worked on our hybrid OCR model, right?
18:29Um and so we have a specific OCR uh like you know configuration that you can enable here that will pri figure out kind of intelligently what to prioritize, right?
18:39Um so in some cases we will actually pick out like just you know strict OCR data or we might pick out like um you know metadata from the PDF.
18:48Um yeah we've done a lot of work here.
18:50Uh and it's it's not a simple sol like it's not a simple answer.
18:54Um this is something we could do an entire different talk on.
18:59Yeah.
19:23So I'll be frank with you like this is actually exactly what this pipeline is doing, right?
19:27So like we first parse the like the Hold on, let me turn return figures.
19:35Cool.
19:36So, um, while I rerun this, what we're doing here, let me pick a different example.
19:42Ah, actually, we're gonna have to wait now.
19:44But what we're actually doing in like extraction is taking the parse results and then feeding them into, you know, some extraction pipeline downstream.
19:53So, like parse is kind of like the um, you know, it it is it is what made reductto reductive in the first place, right?
20:00And then we can do a lot of cool things, you know, downstream of reductto.
20:04So uh you know once we have that parse content for example right we could split you know the document into different regions and figure out like if I'm parsing some you know a math textbook I want to understand like where exactly
20:18every chapter you know starts and ends and then for each chapter I might want to do like an extraction based on that right and uh get you know the chapter summary or something like that right
20:28uh so parse is really the kind of foundation for all of these other agentic um document processing u features.
20:43Man, if I answer that question, I might lose my job tomorrow.
20:47[laughter]
20:48So, um I mean it's a Yeah, continue.
20:52I think it depends on like what pro data we collect um as well.
20:58So we specifically um train on like certain types of data.
21:02If we find that it's doing poorly on certain types of data.
21:06So we see that like long context recall is not great.
21:09Then we got to like try and fix that.
21:11Specifically for extract um it's it's not really like Neil in a haststack.
21:16It's like all the needles in a haststack if you think about it.
21:19So that's like a really um challenging like research problem that our MLES are like somehow able to figure out pretty well.
21:26But yeah, happy to chat more later.
21:30Yeah.
21:31Oh, we've been neglecting you guys.
21:32Yeah.
21:32What's up?
21:33You you you'll be Yeah, you should you should stay tapped into our uh like our our socials and you'll find out about this very soon.
21:40I think like benchmarking is you know always a topic of discussion and something that we want to really stay on top of especially because you know somebody who is testing who wants to do document extraction
21:52might not you know know where to go or who's better right without actually testing themselves.
21:58Uh so that's why benchmarking is like something that's you know we're we're prioritizing a lot uh especially in the coming months to to try to show people like how well we are doing.
22:07you know, we try we're we're very confident in our models and us being like, you know, the premier document extraction solution on the market.
22:13Uh but there need to be uh more, you know, um data points that we can rest on and that the, you know, entire industry can kind of see at face value
22:24uh without having to go test themselves or pass in the all entire ARC AGI data set to you know try to understand uh you know how well reducto is doing against another solution.
22:34Yeah.
22:35So we're going to release some benchmarks soon.
22:37But the real deeper answer here is you typically have to make your own benchmarks unfortunately.
22:43So every single one of our customers typically have some benchmark.
22:46They're like, "Oh, we're evaluating document providers.
22:49Uh we're going to run some head-to-heads uh of like reducto versus like all these other like use cases and you know sometimes they don't use like maybe our APIs really well because they use the wrong config.
23:00So we work closely with them like hey turn this on turn this on you get this trade-off and then uh we let them decide um and then like we often like win these headto-heads in like performance.
23:11So as with most projects you know gather a simple data set you can do some like vibe eval just look look in studio see if the outputs make sense and then like as you want to scale it out then get like 100 documents a thousand documents do something more systematic
23:26and you can also work with our team where we have a lot of people to you know work with customers to help you do your emails but yeah
23:34cooly um any more questions yeah sure well yeah absolutely right um I I think so I'll answer this uh pretty honestly right I think when I was working on this solution right or I was working on this problem I was trying to figure out like what is a good productized
23:52way to you know surface our reducts parsing right um and to be frank with you right like I came up with something that was pretty shoddy in the beginning
24:02um like basically a you know rag chatbot application where you can ask anything about the emails right and it might be something that would be really really useful for like a journal journalist, right?
24:11However, um I think it's testament to the hundreds of millions of people that you know were, you know, using Gmail and and the tens of million that still, you know, log on uh from week to week nowadays.
24:24Um it's it's testament to the fact that a nice product on top of a really really nice novel, you know, uh document extraction solution is what is going to kind of be a onetwo punch here.
24:35Yeah.
24:35Like would you read three million like scan documents?
24:39Honestly, no. And honestly, when I saw the drop, I was like, "This is like way too much work for me."
24:44But once Omar built it, I just like logged on and like started searching like random like famous people for fun.
24:50And so that's the beauty of it, right?
24:52Like I think like unstructured data is like everywhere.
24:55And sometimes all you need to take all you need to do is like to extract it and then like do whatever you want with it.
25:01It's basically what all our customers like do um conceptually.
25:05So I think Gmail is a pretty fun example but it's actually very u it's exactly what happens in the industry.
25:11Yeah.
25:12You seen that?
25:13Come here.
25:14It's it's just hard work, right?
25:16Like you know we we have a data labeling team, right?
25:19We have to go like par you know like we have to sift through a lot of documents if we're going to train a new model on top of it.
25:25Um and be very very certain in the outcomes of you know or or the the sanctity of that data, right?
25:31Um and so the short answer for you is just
25:34[snorts]
25:34like it's a lot of hard menial annoying work, right?
25:38Uh but that's really what comes into like the beauty of a machine learning model at the end of the day, right?
25:43There's nothing that you know really beats that like really really hard, you know, just wrote classification work uh that then compounds into uh model training and then you know then you can kind of figure out like how to stagger your data and like how to train you know on on top of your training set.
25:59At the end of day, at the end of the day, you still have to label that data.
26:03Cool.
26:04I think we'll take Yeah, we could take one more, but um Yeah.
26:08Yeah.
26:08So, to repeat the question uh and let me know, you know, how how accurate it was, right?
26:12But, uh it seems like, you know, like there are cases in which uh you need to really customize your logic on top of different documents.
26:21Uh whether it be corresponding to specific models, right?
26:24Sometimes there might be metadata set up in specific ways um in specific documents and you want to really like uh condition um for these specific use cases, right?
26:34Um this is a fantastic question, right?
26:37And so this is where our agentic um section really comes in, right?
26:41So you can actually apply all of these different, you know, uh, configs and apply like, uh, system prompts on top of separate models that will do specific processing that's tailored directly to what you're asking for.
26:54So for example, right, if uh you want to do some complex figure extraction, let's say I have a bunch of logos and I want to try to determine like you know what differentiates
27:05like I don't know uh some you know uh Lincoln Financial from like Google and like how to separate those kinds of logos, right?
27:13I could add some additional color here within our uh you know prompts uh in this config to custom kind of tailor u the outputs that I'm looking for.
27:24So if you wanted some specific metadata here, let's say like uh you know some layout might be like really really hard to read in uh I don't know like a construction blueprint, right?
27:34I would spend a lot of time in the uh layout prompt kind of tailoring to try to understand uh how well you know uh my current extraction solution is mapping across all of my different
27:46um you know construction blueprints I guess right cool.
27:50Um does that answer your question?
27:52Fantastic.
27:53All right.
27:53So now we want to kind of um spin this off um into an interactive exercise.
28:02Right.
28:02So, um, if you guys would like, we're going to basically give you guys, you know, um, however much time you would like.
28:09I think we have about 30 minutes left.
28:11Um, and you could sign up today at, uh, at studio and essentially, uh, just get started, uh, on some document extraction solution, right?
28:20So, what what you'll see here in a QR code is something that you could basically paste directly into a cloud code or a codeex or whatever, and something you could just get started
28:28uh, on uh, document extraction with immediately.
28:31Uh so we're going to give you guys about 30 minutes and by you know maybe around like five minutes uh like at the five minute remaining mark we'd love to kind of see what you guys have cook uh cooked up and we have a bunch of cool swag and a couple of cool prizes for you guys
28:45uh to just kind of show us what you got um with respect to document extraction.
28:49Um it could totally be just like an idea um and like a singleshot prompt but uh feel free to just kind of uh give it a whirl.
28:57Sound good?
28:58Also, we'll be kind of like circling around if you guys have any questions as to how to use Redto.
29:03Uh, I want to come work for you guys, you know, I want to hear about your models, how you guys train, whatever.
29:08Um, we are happy to answer.
29:10Sound good?
29:26[music]