Better Search Took This AI Agent From 36% to 98% Accuracy - Aamir Shakir, Mixedbread

AI Engineer · 16 min · 137 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12So quickly about mixread and what we're doing before I jump into the interesting things.
00:17So mixread started as a research lab around retrieval since we were really sad about the state how the space was evolving.
00:24So we have been doing research for the two and a half years over how we can make retrieval really really good and we come ended up with a product where you can ingest any kind of data and make it searchable for agents meaning PDF,
00:36spreadsheets, images, video, audio in over 100 languages.
00:40So it's a true multimodel multilingual search where we are owning the models.
00:45So we built the models from scratch specifically for search.
00:48We build our own database so we can provide zero data retention and all the privacy privacy requirements you need and also we build injection pipeline so you don't have to hustle to managing injection managing OCR and knowing what to do and instead you just dump your data into it rely on our models working good enough
01:05and you know so you can build really really powerful agents and I'm really happy that over the last three months like we're live since 5 months but over the last three months we onboarded banks and hedge funds into mixread
01:17and alive with it with billions of documents and the main reason is we are so ISO compliant and we have this bring your own bucket mode meaning that we can really
01:26offer all the security requirements you need with zero data retention so when we talk about evidence layer or using really really powerful agents right it mostly comes down to
01:36working on your data so if you have like oh sorry um how to move this forward so assume that you're building an agent or you have an equity analyst
01:46this is just like an example query because Intel yesterday released their new GPU cards, right?
01:51Like the D270 and somehow it has a really weird design that it's using a lot of LPDDR5 memory.
01:58So, you know, like you're an analyst sitting there and what the Why is Intel publishing this, right?
02:02Why why does it have so much LPDDR5 and not HBM memory?
02:05So, it would ask your system a question like, hey, why does the new Intel D270 use LDR5 memory?
02:12And since you guys probably as you're working in finance, you have a lot of data brokers and data providers.
02:17They send you the podcast, they send you interviews, they send you materials, research uh stuff and so on and so forth.
02:23So you have this huge knowledge base and now your analyst is asking this question, right?
02:27So when you use cloud code or cloud cowwork, what it normally does when you have your ingest data locally, it uses the terminal and basically uh lexical search like grab
02:39to figure out, hey, where do we find terms like ADDPR5 or Intel and GPUs to just figure out, hey, what documents can be the closest and the most relevant ones?
02:50And as you have a lot of PDFs and a lot of documents and interviews, you will get a lot of hits and you have no relevancy and you don't know where to look into.
03:00So what the model will end up doing is reading every document and like this, you know, like your token cost will just go up and up and up and the more documents you have, you will end up with per query costing 10 or $20
03:12to just ask a really basic question.
03:14And the second downside of that is also that most PDFs sadly are not really grapable because you know like there are binary formats sometimes there are scans in there images in there so you can't really exploit this multimodel
03:26data relying in there to work for you.
03:30So these are like the key bottlenecks and then you think okay what are people doing and what people have done previously in the spade space who's familiar with the term rack retrie augmented generation just hands up really quickly.
03:42So I think most people so uh I think you will it will resonate what I'm going to say now.
03:48So then we were like okay good let's let's build a sophisticated pipeline where can make all this data really really usable for us.
03:55So we have this billions of PDFs podcast whatsoever.
03:58So first thing we do is we do parsing right.
04:00So if you have um audio we do transcription um if you have PDF we do OCR and what every of the step does it introduces noise.
04:10So if you can if you don't process tables properly, if you don't pass diagrams properly, what will happen is that you will just lose information and then you do the chunking.
04:21So you already lost information because your table is not passed properly or characters are not processed properly, then you do some chunking, right?
04:27So if you have figures, table, pages, where's really the chunk boundary?
04:31So you're introducing more noise and then you're giving it to a pure embedding model which you know like takes this information and then tries to guess a point in the space
04:40to you know like what this information represents and since it already lost so much information you are losing then having this point right and this is then a super lossy process from going from this really rich information to hey here's just a point
04:55and then you put it into a vector database do some retrieval and rerank.
04:59So since every step is lossy and you have to add a vendor for every process, you will end up hey what is the best OCR?
05:08What is the best chunking strategy?
05:09What is the best embedding model?
05:11Uh what is the best vector database?
05:12What is the best ranking?
05:14So you will just evaluate it for six or eight months different vendors run through security to just go live and you have to allocate four to five people to that to just build something from my perspective super basic which won't work in the end because there's some inherent limits of how this
05:31things are done and we internally call it the Frankenstein stack and there like some proofs why you know like the single vector models or like just putting all this information into
05:42embedding model and one point actually doesn't really work and we can discuss it later if you guys are interested why and I can go on a rant and explain it a bit more in depth but this is actually right now the state-of-the-art what people use and this is also the reason like all the bottlenecks I described right
05:59now why rack actually never worked so you know like I think you all remember this promise of hey here's spine cone or here's any vector database just put it in and it will just work but it never really worked out and it comes down to that
06:13every step is super lossy and that also the models we have the embedding models with the technologies they're using at the moment are not good enough so you you can't really use this information properly
06:26and the downside when I talk about this embedding models is you have to imagine like how this embedding model work right you have an information like I love bread you put it into a model and then it just takes this sentence and puts it somewhere in the space and later if you ask hey who loves bread
06:42we'll put also into the model and we'll put it somewhere into space as well and then just search around the query.
06:47Hey, where's the similar point to basically describe that but now let's think about a financial report like a sec filling.
06:54If you have like complex tables with all this different information like net income, revenue yada yada yada what does this point actually represent?
07:03Right?
07:04So maybe it uh represents information about Nvidia.
07:07So it's Nvidia's earnings report, right?
07:09But if you have the earnings report from 1996 or 2000 and 2026 the points will look really really similar.
07:15That's why when you have a lot of information which look very similar rack and all this vector database approaches don't work and we see with a lot of customers
07:24in their classic approach if you have just few documents let's say tens of hundreds of documents the quality is pretty good but as you start scaling and going into production
07:33the answer quality will just degrade significantly because this type of vector search and this type of technology is not able to really distinguish between this fine grained information and the issue is you have a pure
07:46a poor agent, right, who wants to really help you.
07:48He he will do everything his to serve you and do everything for you.
07:52So what will this agent start doing?
07:53It will just query aggressively.
07:55It will spin up tens of sub agents.
07:57Um we'll do more and more queries.
07:59So you know it will just fill up the context window and every query will become expensive.
08:05You can answer it like this by just putting a lot of agents on that.
08:08But the question is hey is it really the way you want to go into production?
08:12If you also imagine at one point that maybe your fund or your VC will be mostly run by agents and the volume of agent traffic will just increase.
08:20So the cost of running the systems will just skyrocket.
08:26So this is why we came to the conclusion pretty early when we're working with our customers that intelligence is there but the real real bottleneck is actually access to your data and to your knowledge
08:39and we believe that we will see a unbundling of intelligence and data where you have intelligence where you can choose between different intelligence models but you need to have the right access for this intelligence
08:50for your data so you can make the most of that and for this we built um mix bread.
08:57So as I mentioned mixread started as a research lab and this is a really interesting graphic and this is a preview of what's coming next week.
09:04Um is that retrieval how it changed like over the last from 1994 to now it's like what 32 years I think my if my mental math is not wrong.
09:14So we started with BM25.
09:15This is what elastic search is doing like simple keyword matching and then we just started coming up with smarter method.
09:21So the first really good embedding models were published in 2022.
09:25So it took the space 27 years to come something up for text search better than BM25.
09:31But since then we have more like a linear um like improvement.
09:36And this is also what we meant with rack not working because there was never a step function change like we saw with chatbt and so and so forth and
09:44we published our first model by beginning uh we finished the training at end of last year published it beginning of this year it was pretty solid but the thing we're publishing next week and we already live with a lot of our customers
09:56we think is a step function change which overcomes most of the challenges I described before just in short what this model does is it's a late interaction model Meaning instead of having just one embedding, you have multiple embeddings for your documents and you have multiple embeddings for your queries like this.
10:12You can really describe fine grained information on a page.
10:15So going back to the SEC filling for Nvidia for different years, right?
10:20So you would have the model would figure out an embedding for net income for net revenue etc etc but since it's like creating it in one pass it also know it's about Nvidia.
10:29So every embedding is contextualized and with the information which are relevant to you and all this technique and scaling up the model and better data gives us significant jump in quality
10:40uh which allows to do some cool things which allows to actually build this product uh mix where you just upload your data mix takes care of indexing it for you like every page
10:51as it looks because it's multimodel I will show what it means and you then just can give it agent access to mixread and the agent does the querying the reading the grabbing everything for you and you can just focus on on what's fun actually building the real application
11:04and actually building real RARI right so um let me show you what it means so this is mixre and what I did here is I uploaded some um Intel
11:18um wait hold on where is it exactly so I uploaded some Intel reports here for example the Intel earning stack so what you do in mix is just upload the uh page.
11:30The internet is pretty slow.
11:31I
11:31[snorts]
11:31hope it will be fine.
11:34So, it's it's failed to load because the internet is slow here.
11:37I'm so sorry about that.
11:39Um I should have recorded a video.
11:40Um so, let's do this.
11:45Cool.
11:46So, what it does here is so we have a slide.
11:49Come on, dude.
11:50So, you see there's a uh is there's a figure, right?
11:53So normally if you would just do OCR and the figure and you want to know the actually the revenue of Intel in Q2 2025 if you just would do OCR you would just all lose all information it would never really able to match the page because you need to understand
12:07what's going on there but if you create the embedding on the visual perspection like looking let the model look at the page the screenshot and create the embedding based on that you are getting rid of all this useless stuff I was talking before
12:19but since not all LM are multimodel mixread still provides stuff like layout information and OCR for your downstream thing where you can decide to put in the screenshot or the text yourself to do the right thing and let's just try it out together.
12:33So let's say we want to know hey what was in Intel's foundry segment operation mar mar mar mar mar mar mar mar mar mar mar mar mar mar mar mar mar mar mar mar margin in Q2 2025
12:41it goes out I hope it will work now normally it's like sub 200 milliseconds but you saw the thing before um so we see hey it finds somehow this page here which is talking about in Foundry
12:54and down here come on you can do it here you go it's talking about Q2 to 2025 and it understands also hey it's somehow about segment revenue and can really pass all this information
13:09and since we have this visual layout stuff we can actually reference this with bounding boxes and when your analyst or your agent is deriving to a decision you can actually backtrack and site where it is coming from and we're providing all the APIs and all the primitives for you to build this experience yourself
13:25and if you have like really really complex queries for example you want to know like something more qualitative Hey uh is Intel's foundry uh prof uh profitably improving and what is driving it?
13:40So if you just query it for example our system realize hey this is a more complex query let's do a genetic search a genetic search is a model we trained toast one it's like a
13:51agent which does multiple queries for you multihop reasoning and it's as good as Astra and fable while being 40 times faster and cheaper which then goes out and does like multiple queries
14:02um and can really decompose everything and come back with the right data and uh this data can then fed into an LLM to actually explain what what what is happening there and we just also expose it as a primitive.
14:14This is just a demo, a playground to show you what's possible, but you can just use it, give it to an agent and let the agent decide if you want to use grab, normal search or the agentic search depending on the complexity.
14:25Exactly.
14:26And yeah, um so this is uh was as a backup if it wouldn't have worked.
14:33Um so the agent can do everything.
14:34And here a small case study which we will publish also in two weeks from now.
14:38It's like a really big hedge fund.
14:39We on boarded within like one and a half weeks.
14:42um because they have like over 1 billion documents right now and they are building a lot of internal agent platform for the analyst and for their fund managers to derive qualitative
14:52uh analysis and also come up with should we is it a good decision or not and before they have built what I described before this chunking betting OCR some random embedding model some vector database
15:04and they had a answer accuracy like how often the agent is right of 36% where the human experts knew the answer and graded And once they move to mix spread
15:14the answer accuracy moved to 98% because the again the issue is not intelligence it's again getting the right data and just by providing way better search to your agents
15:24you can first make it really good um here's a breakdown on multimodality and text questions and as you can see multimodality is just bumps it quite significantly but second is you use also wafier tokens so the LM spend is going down by 3x
15:39because again most of the tokens are used to find, pass, and read information.
15:44And as you just fit in really good context, you know, like you just need way less LLM cost and the LM can just focus on what's relevant.
15:53Um, exactly.
15:54So, this is my talk in a nutshell.
15:57Um, if you want to try out MixPread, uh, this we have toast, uh, flyers downstairs, so if you want to take them, I think they're pretty funny.
16:05Um, and you can sign up.
16:06We have free credits.
16:07Uh, if you are concerned about security, you can just write us.
16:11We activate bring on bucket within 5 minutes for all of our customers and then you can just start trying it and if you have like really large corpuses we provide a lot of free credits so you can benchmark it before
16:21really onboarding it.