WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=1IdzkRVmWAA

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/1IdzkRVmWAA.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:00.000 --> 00:00:10.640
Hey, today we're going to talk about how we taught agents to use good retrieval or how we call it internally closing the Oracle gap with knowledge agents.

s2
00:00:10.640 --> 00:00:11.680
I'm Hannah.

s3
00:00:11.680 --> 00:00:17.000
I'm an AI engineer at Mix bread and I'm leading the agentic search at Mix bread.

s4
00:00:17.000 --> 00:00:18.000
Yeah, I'm Amir.

s5
00:00:18.000 --> 00:00:21.480
I'm the co-founder of Mix bread and let's jump into it.

s6
00:00:21.480 --> 00:00:31.240
So, we have seen over the past years with the LLMs just getting better and better that the reasoning capabilities of the models is growing exponentially.

s7
00:00:31.240 --> 00:00:37.480
So, just think back how good GPT-3.5 was and how good GPT-5.5 was, right?

s8
00:00:37.480 --> 00:00:39.480
So, I can clearly exponential curve.

s9
00:00:39.480 --> 00:00:49.720
But, if you look at search over the last 20 years basically, we see that it's improving, but it's improving very very slowly.

s10
00:00:49.720 --> 00:00:59.120
So, there's a huge gap between how basically LLMs and the reasoning is evolving and how retrieval is evolving.

s11
00:00:59.120 --> 00:01:13.080
But, the retrieval tools are the main access pattern for this reasoning there to get the right knowledge and to be truly useful beyond code like in legal work, in finance work and so on and so forth.

s12
00:01:13.080 --> 00:01:18.880
Internally, we call this gap happening right now between reasoning and search the knowledge gap.

s13
00:01:18.880 --> 00:01:22.240
And it's not just one obscure theory of ours.

s14
00:01:22.240 --> 00:01:28.280
We see that actually with real benchmarks and real tasks that this gap exists in real world as well.

s15
00:01:28.280 --> 00:01:33.080
Let's take two benchmarks here, BroSque Plus and Office QA Pro.

s16
00:01:33.080 --> 00:01:44.400
BroSque Plus is like a browsing benchmark where we have pre-complex queries and a corpus of 100,000 documents and we just try to answer these questions.

s17
00:01:44.400 --> 00:02:00.880
It's like real world deep search task basically and Office QA Pro which has the treasuries of the US treasuries of the past um 100 years basically and we ask really complex question over this.

s18
00:02:00.880 --> 00:02:11.560
Office QA Pro was created by Databricks and BrowseComp was created by OpenAI and BrowseComp Plus is a version of it with a fixed size corpus while BrowseComp was over the open web.

s19
00:02:11.560 --> 00:02:23.640
And we see here's a Oracle performance in dotted lines and Oracle means what is the maximum theoretical performance of the models if you would put in the right documents

s20
00:02:23.640 --> 00:02:25.880
with the question with with the question.

s21
00:02:25.880 --> 00:02:34.200
We see for Office for BrowseComp it's 93% and for Office QA Pro it's 64%.

s22
00:02:34.200 --> 00:02:42.160
And now we take something like Codex with its default tools and want to see hey, what is the performance this tools get right now?

s23
00:02:42.160 --> 00:02:48.760
And you see there's a sharp drop in the quality of the answers Codex produces.

s24
00:02:48.760 --> 00:02:53.000
For BrowseComp it's nine points and for Office QA Pro it's eight points.

s25
00:02:53.000 --> 00:03:05.920
So we see that the models are extremely capable if they would get the right documents but if you put them into the noisy corpus the performance drops sharply.

s26
00:03:05.920 --> 00:03:11.400
Meaning that actually the bottleneck here is not the reasoning.

s27
00:03:11.400 --> 00:03:16.600
It's actually the access to the right knowledge it needs to answer this question.

s28
00:03:16.600 --> 00:03:20.920
So we see this knowledge gap in real world.

s29
00:03:21.440 --> 00:03:30.640
And if you would just drop in way better search using Mixbread with just a search tool using latent interaction we can recover most of the performance.

s30
00:03:30.640 --> 00:03:41.600
So for BrowseComp the difference between the Oracle and the uh GPT-5 with Mixbread is just three point and for Office QA Pro we even almost completely closed the gap.

s31
00:03:41.600 --> 00:03:47.120
With giving the model better search tool we can recover most of the knowledge gap.

s32
00:03:47.120 --> 00:03:55.800
But looking at the queries the model asks or writes right now to the search system, it gets super super interesting.

s33
00:03:55.800 --> 00:04:07.120
So, here's an example query we found uh during some benchmarking, which is senator woman questions billionaires not a company then okay thank you staff will check hearing.

s34
00:04:07.120 --> 00:04:08.520
It's basically gibberish, right?

s35
00:04:08.520 --> 00:04:18.120
If you have a search system which wants to have neural questions or semantic question and just gets these keywords over keywords, it gets confused.

s36
00:04:18.120 --> 00:04:29.360
And the reason why the models write these type of queries is they're mostly trained for coding task, like for coding agents, which are then optimized for code base exploration

s37
00:04:29.360 --> 00:04:31.680
using tools like grep.

s38
00:04:31.680 --> 00:04:36.000
And grep just tries to find regular expressions, right?

s39
00:04:36.000 --> 00:04:41.960
So, the model tries to just write as many expression it thinks are in the document to find the right thing.

s40
00:04:41.960 --> 00:04:47.040
The second thing is the models are trained to use the web pretty efficiently.

s41
00:04:47.040 --> 00:04:55.120
And to use the web tools properly, which are highly optimized for humans, they try to mimic human-like query patterns.

s42
00:04:55.120 --> 00:04:57.480
So, just keyword on keyword on keyword.

s43
00:04:57.480 --> 00:05:00.400
And number three, obviously, is the benchmark bias.

s44
00:05:00.400 --> 00:05:11.760
Most benchmarks we have right now, like Beer, Nano Beer, use Caveman's style queries, which are entity-based queries that structurally favor heavily BM25.

s45
00:05:11.760 --> 00:05:23.200
So, right now, the agent guesses the keywords to actually increase the overlap between the query and documents and can't really use powerful search tools properly.

s46
00:05:23.200 --> 00:05:35.760
This motivated us to make our own search agent, to teach it to use powerful search properly, especially to use the right search tools for different use cases.

s47
00:05:35.760 --> 00:05:40.200
Um and to work beyond code search for knowledge work.

s48
00:05:40.200 --> 00:05:44.440
And most importantly, the agent should be precise, fast, and cheap.

s49
00:05:44.440 --> 00:05:50.120
The first step for building this agent was to define a very powerful harness.

s50
00:05:50.120 --> 00:05:53.880
Um [clears throat] our harness is fully built on a mixed back platform.

s51
00:05:53.880 --> 00:05:59.240
Our agent has four main search tools, which are overview search.

s52
00:05:59.240 --> 00:06:07.080
This is used as a very wide semantic search where the agent receives up to 50 retrieve chunks.

s53
00:06:07.080 --> 00:06:19.880
Um and it sees only summaries of the chunk contents to really have just like an overview of the corpus every uh what exists and to not fill up the context too much.

s54
00:06:19.880 --> 00:06:27.840
Then there is the main semantic search tool where the agent gets the full payload of the top 10 retrieve chunks.

s55
00:06:27.840 --> 00:06:36.200
Um it has a filter chunks tool where it can sort sort and find chunks based on metadata facets.

s56
00:06:36.200 --> 00:06:41.000
And of course we also have a grep tool for the keyword match searches.

s57
00:06:41.000 --> 00:06:48.840
Our um agent loop, the harness itself is very simple and short cuz the agent should be fast.

s58
00:06:48.840 --> 00:06:54.800
But we still wanted it to uh have a lot of exploration possibilities.

s59
00:06:54.800 --> 00:07:06.720
That's why we decided to um define that it has a maximum four search rounds, but within each search round it can have parallel searches.

s60
00:07:06.720 --> 00:07:10.720
So it can start several search at that searches at once.

s61
00:07:10.720 --> 00:07:25.360
Um initially our agent sees the user query, but also already um results from an initial semantic search on the user query plus um hints on which metadata facets are available.

s62
00:07:25.360 --> 00:07:31.680
Um so that based on this preview of the corpus the agent can start its search planning.

s63
00:07:31.680 --> 00:07:41.520
It splits its search intent into maximum four queries um which cover separate aspects.

s64
00:07:41.520 --> 00:07:49.000
And then for each query and each search intent, it will pick the best tool.

s65
00:07:49.000 --> 00:07:53.000
Um as you can see like an example here on the right.

s66
00:07:53.000 --> 00:08:06.670
Then when um the results of the tools are returned to the agent via deduplicating chunks, so that it's never seeing several chunks um several times to not fill up the context.

s67
00:08:06.670 --> 00:08:07.240
[snorts]

s68
00:08:07.240 --> 00:08:17.840
Um and yeah, the next search one can start, and as soon as the agent has enough evidence to answer the user query, it submits its ranking.

s69
00:08:17.840 --> 00:08:25.440
Um yeah, we are just outputs all the chunks that are relevant to the query in a plausible ranking.

s70
00:08:25.440 --> 00:08:32.640
So, why does this harness encourages better semantics queries and like better use of search tools?

s71
00:08:32.640 --> 00:08:38.080
Um there are five more main points for this, which is first the goal framing.

s72
00:08:38.080 --> 00:08:45.280
The agent has to articulate articulate what evidence it needs before writing the query.

s73
00:08:45.280 --> 00:08:58.680
Um then the different tools are very important, so it can really use semantic search only if it needs aspects, and also only use a scrap when it needs exact keyword matching.

s74
00:08:58.720 --> 00:09:07.080
Um then the way in which we frame the task of query uh writing is an important point.

s75
00:09:07.080 --> 00:09:21.120
We kind of trick the um model into not thinking it has to write the typical BM25 base query by just instructing it to write one concise sentence describing what it wants to find,

s76
00:09:21.120 --> 00:09:27.600
instead of directly instructing write a search query, so it cannot fall into this old pattern.

s77
00:09:27.600 --> 00:09:41.280
Um then we put also provide examples in a prompt just to show the a few good queries and also how to divide an input query into different aspects to explore

s78
00:09:41.280 --> 00:09:43.520
um the corpus.

s79
00:09:43.520 --> 00:09:44.480
Yeah.

s80
00:09:44.480 --> 00:09:51.280
What also helps is that we provide the original original query semantic search results.

s81
00:09:51.280 --> 00:10:01.440
So, it can see what the corpus is about and really define um see the language, define where to dig deeper in which aspect.

s82
00:10:01.440 --> 00:10:08.480
The second important step into building our own search agent was of course the training itself.

s83
00:10:08.480 --> 00:10:16.040
So, here for training, we just decided to have a very small LLM to make the agent even faster.

s84
00:10:16.040 --> 00:10:31.560
So, we are training the small agent um to optimize the search strategy itself to have better tool choice, um higher quality of semantic queries, more exploration ranking, and to be more efficient.

s85
00:10:31.560 --> 00:10:37.760
So, the first step in training is um supervised fine-tuning with a larger teacher LLM.

s86
00:10:37.760 --> 00:10:45.560
And then we uh did on-policy reinforcement learning with an own search reward.

s87
00:10:45.560 --> 00:10:54.280
Our search reward is a combination of both of a retrieval reward and a trajectory reward.

s88
00:10:54.280 --> 00:11:08.160
The retrieval reward um is based on like the metric result, typical retrieval metric results like we call it NDCG, that's the agent's final ranked list achieves.

s89
00:11:08.160 --> 00:11:13.080
Plus also um an LLM retrieval judging.

s90
00:11:13.080 --> 00:11:24.960
We had thought like it gets several rubrics where it decides um if the rubrics is hit or not, the rubric is hit or not, which are if the agent result relevant for this query,

s91
00:11:24.960 --> 00:11:31.080
are all chunks relevant, and is the ranking itself plausible.

s92
00:11:31.080 --> 00:11:42.880
For the trajectory reward, we also using an LLM judge and here um is the point to like really improve the tool queries, make it more efficient and to um

s93
00:11:42.880 --> 00:11:46.120
improve the quality of the queries.

s94
00:11:46.120 --> 00:11:53.200
Of course, like we have rubrics that are where the judge is deciding if the query is really a natural sentence.

s95
00:11:53.200 --> 00:11:59.600
Also, if the amount of exploration is sufficient, is it too much or too less.

s96
00:11:59.600 --> 00:12:03.520
Here you can see an example trajectory.

s97
00:12:03.960 --> 00:12:09.440
You see that in the beginning we have the initial search and the initial metadata hints.

s98
00:12:09.440 --> 00:12:16.880
Um then the agent did four parallel searches in the first round and then a second round just a simple grep tool.

s99
00:12:16.880 --> 00:12:21.200
And here's an extract of um a trajectory.

s100
00:12:21.200 --> 00:12:24.760
Here we see the agent queries.

s101
00:12:24.760 --> 00:12:27.800
Um the input query is on the left.

s102
00:12:27.800 --> 00:12:35.960
This is a very long query typical rambling style query from the Oblique Congress benchmark.

s103
00:12:35.960 --> 00:12:40.880
And yeah, we see the agent queries here for the first semantic search tool.

s104
00:12:40.880 --> 00:12:51.600
It's really a sentence describing what it wants to find and not a weird keywordy added behind each other formulation.

s105
00:12:52.280 --> 00:12:58.640
The grep tools of course are like typical keyword patterns, which is exactly as intended.

s106
00:12:58.640 --> 00:13:02.200
Our trained agent is not released yet, unfortunately.

s107
00:13:02.200 --> 00:13:05.640
However, we have some intermediate results.

s108
00:13:05.640 --> 00:13:24.240
Here on the left side for the Oblique Congress benchmark, we see that we achieved an NDCG of 10 at 10 of 0.4, which is a huge jump um towards the model that performed best on the paper of this benchmark, which is the GPT multi-hop agent,

s109
00:13:24.240 --> 00:13:28.280
and which achieves 0. 18.

s110
00:13:28.600 --> 00:13:36.160
On the right side, you can see the result of our beta version of the search agent, which we have right now in production.

s111
00:13:36.160 --> 00:13:39.320
It's the mixed better genetic search.

s112
00:13:39.320 --> 00:13:55.440
And this agent is top one on the snowflakes match QA benchmark, achieving an accuracy of 93.4 when we give the Gemini 3.5 flash model our genetic search as search tool.

s113
00:13:55.440 --> 00:14:06.360
And this uh this performance was also achieved while having way less effort than comparable LLMs with other search agents.

s114
00:14:06.360 --> 00:14:09.400
You can check out the match QA leaderboard.

s115
00:14:09.400 --> 00:14:18.160
These results show that there's still a lot of room for improvement when it comes to huge language models and their search tools.

s116
00:14:18.160 --> 00:14:27.160
And if you want to be part of pushing the boundaries of retrieval even further, um we are happy awaiting your application.