WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=fZH97QHHYjY NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/fZH97QHHYjY.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:01.309 --> 00:00:03.309 [music] s2 00:00:12.639 --> 00:00:14.000 So great being here. s3 00:00:14.000 --> 00:00:15.599 Uh I'm Joe Bergam. s4 00:00:15.599 --> 00:00:23.600 I'm the CEO of Hornet Dev and I'm here today to talk about the unreasonable effectiveness of BM25 for Agentic Search. s5 00:00:23.600 --> 00:00:26.000 So how many of you heard about BM25 before? s6 00:00:26.000 --> 00:00:28.560 Is it new or is it Well, quite a few. s7 00:00:28.560 --> 00:00:29.519 So that's great. s8 00:00:29.519 --> 00:00:32.160 Uh I'm also watching the World Cup. s9 00:00:32.160 --> 00:00:35.840 Norway is playing there uh against the Ivory Coast second half. s10 00:00:35.840 --> 00:00:38.800 Norway is leading so that's good. s11 00:00:38.800 --> 00:00:49.440 So yeah and at Hornet we are building retrieval infrastructure for agents and uh I've been working on search and retrieval problems for a long time. s12 00:00:49.440 --> 00:00:51.680 As you can tell I'm gray hair. s13 00:00:51.680 --> 00:00:56.079 Uh been working in this space for more than 20 years. s14 00:00:56.079 --> 00:01:06.799 Um and in the talk today I'll talk about why this kind of 30-year-old lexical scoring function is uh making a strong comeback. s15 00:01:07.119 --> 00:01:14.799 First I will talk a little bit about what I mean by agentic search or aentic dribble and kind of define that for for you all of you. s16 00:01:14.799 --> 00:01:22.400 Um my definition is that agentic search is essentially search inside agent loop. s17 00:01:22.400 --> 00:01:34.400 So you have an agent that is trying to accomplish a task, write a coding or do some deep research or whatever task and inside that you have some kind of information need for the agent, right? s18 00:01:34.400 --> 00:01:49.040 In order to get that task done successfully and you essentially need three things uh to kind of build a good agentic search system and that is you need a capable model, a model that is able to use tools s19 00:01:49.040 --> 00:01:51.920 and is able to formulate queries. s20 00:01:51.920 --> 00:02:00.640 You also need a harness around the model and how you kind of expose the retrieval and the search functions to the model. s21 00:02:00.640 --> 00:02:02.240 There's different ways to do that. s22 00:02:02.240 --> 00:02:06.240 It could be through tool calling or it could be through code mode. s23 00:02:06.240 --> 00:02:13.680 Um, Edo here uh demonstrated what I call code mode for exposing retrieval infrastructure. s24 00:02:13.680 --> 00:02:29.280 So that's the harness part and you also need uh a retrieval engine to be able to do searches efficiently potentially over billion scaled uh document sets and to define BM25. s25 00:02:29.280 --> 00:02:33.040 So BM25 actually stands for best match 25. s26 00:02:33.040 --> 00:02:40.560 So there were some researchers doing plenty of different experiments and experiment number 25 turned out to be the best one. s27 00:02:40.560 --> 00:02:43.440 So that's that's the background of the name. s28 00:02:43.440 --> 00:02:45.040 It's essentially a scoring function. s29 00:02:45.040 --> 00:03:02.560 So you can imagine you have a query and you have a document and you calculate some kind of score by interaction between the query terms and the document terms and you come up with a score and you hope that this score kind of is a good proxy for the relevancy of the document with regards to the query. s30 00:03:02.560 --> 00:03:02.879 Right? s31 00:03:02.879 --> 00:03:12.720 So one way to calculate BM35 would be to take all the documents and score each of them you know and then figure out what are the top K documents s32 00:03:12.720 --> 00:03:23.519 and then there's like 30 or 40 years of interest in how you accelerate that type of retrieval top K lots of different algorithms we also invest a lot in that I'll s33 00:03:23.519 --> 00:03:35.360 show something in this direction but BM25 is the scoring function and there is a way to kind of accelerate top K retrieval Uh BM25 hasn't changed. s34 00:03:35.360 --> 00:03:41.280 It's the same scoring function, but the change here is really that we got a more powerful user. s35 00:03:41.280 --> 00:03:44.159 Uh Edo talked about general knowledge. s36 00:03:44.159 --> 00:03:46.799 The LLMs today have a lot of general knowledge. s37 00:03:46.799 --> 00:03:52.879 They know about entities, they know about companies, they know about dates, they know a lot, right? s38 00:03:52.879 --> 00:04:01.840 And by using that kind of implicit knowledge that is built into the parametric model, uh they essentially become very good at search. s39 00:04:01.840 --> 00:04:13.920 Um and that's the really change uh here now that we're is kind of making BM25 more relevant and BM25 used to be a kind of a baseline function. s40 00:04:13.920 --> 00:04:25.040 Any information retrieval research would include a BM25 baseline and then you would put something fancy advanced neural fancy stuff and then you would compare it with BM25. s41 00:04:25.040 --> 00:04:34.639 I also think it's interesting in how we evaluate search before because you will simply look at 10 blue links and you like have scan it and you compute some metrics. s42 00:04:34.639 --> 00:04:45.280 A lot of that is now going away because the agent is not really it's kind of powerful in the way that it can type out a lot more queries than a human can do. s43 00:04:45.280 --> 00:04:52.000 So it's less relevant to think about evaluating these systems just by a single shot query. s44 00:04:53.840 --> 00:04:58.080 And this is one of my kind of favorite benchmarks out there. s45 00:04:58.080 --> 00:04:59.520 I like to talk about benchmarks. s46 00:04:59.520 --> 00:05:03.923 So browsecom plus is a deep research benchmark s47 00:05:03.923 --> 00:05:04.479 [snorts] s48 00:05:04.479 --> 00:05:14.160 uh published in a paper last year and it has almost or exactly 830 questions. s49 00:05:14.160 --> 00:05:17.360 These are riddle like think about it like a pub quiz. s50 00:05:17.360 --> 00:05:19.440 Do you have pub quizzes in the US? s51 00:05:19.440 --> 00:05:19.759 Yeah. s52 00:05:19.759 --> 00:05:20.479 Okay great. s53 00:05:20.479 --> 00:05:31.600 So like kind of a riddle type of questions quite long and the agent the harness of this kind of or the protocol of this benchmark is that you have a model s54 00:05:31.600 --> 00:05:43.199 and it gets a very simple tool called search and it accepts a query string and you return some snippets back to the model and the corpus is about 105,000 s55 00:05:43.199 --> 00:05:44.560 or 100,000 documents. s56 00:05:44.560 --> 00:05:50.320 So that's kind of tiny and these are web documents and the end to end accuracy. s57 00:05:50.320 --> 00:05:59.840 Um all of these questions have a golden reference answer and you can kind of check if the model and the entire loop produces that exact answer. s58 00:06:01.759 --> 00:06:06.639 Um but why kind of why do we need retrieval? s59 00:06:06.639 --> 00:06:11.759 I like to compare context windows with floppy discs because I'm old in the 80s, right? s60 00:06:11.759 --> 00:06:15.840 We install this kind of games on our computers using floppy discs. s61 00:06:15.840 --> 00:06:27.120 So one kind of you guys are so young so you don't probably have this kind of nostalgy but one floppy disc could fit about 1.4 megabytes of data and s62 00:06:27.120 --> 00:06:35.039 the current models before they start degrading in quality that's I in my opinion around 350,000 tokens. s63 00:06:35.039 --> 00:06:38.160 So that's one floppy disc of data right. s64 00:06:38.160 --> 00:06:54.800 Um, so you need retrieval in order to to fetch the information that you actually need to put into the context window and browse plus really demonstrate how uh retrieval quality s65 00:06:54.800 --> 00:06:57.440 affects the end toend accuracy of the task. s66 00:06:57.440 --> 00:06:57.680 Right? s67 00:06:57.680 --> 00:07:05.520 The endto-end accuracy here is essentially is the model equipped with this search tool able to answer the question, right? s68 00:07:05.520 --> 00:07:21.440 the riddle like question and if you artificially just stuff the evidence documents that is needed to answer this question into the context window of the model the accuracy is really high s69 00:07:21.440 --> 00:07:37.520 right so reasoning is not the bottleneck the model given the edit evidence up front answers the question with a very high accuracy rate even GPT4 um but if you expose the model with a harness with a retrieval tool. s70 00:07:37.520 --> 00:07:41.759 That accuracy falls because it now depends on the harness. s71 00:07:41.759 --> 00:07:48.720 It depends on the model's ability to formulate queries and the retrieval quality of the retriever. s72 00:07:48.720 --> 00:08:00.639 Um so for me this is also quite important because even if we get perfect models like models that are kind of agi you don't have to append make no mistakes s73 00:08:00.639 --> 00:08:09.120 you still will be limited to a context window that is approximately a floppy disk right so you have to decide what goes into that context window and I think retrieval s74 00:08:09.120 --> 00:08:24.240 is still very relevant as the previous slide showed and in this browsecom plus data set one of these riddlelike questions become a search trajectory because the model execute a query, s75 00:08:24.240 --> 00:08:35.120 gets some response back, reads it, reformulates the query and continues until it has kind of filled up the context window or found the answer, whatever comes first. s76 00:08:37.120 --> 00:08:52.720 And we spend some time to investigate these trajectories uh to see how GPT5 is formulating queries and we found a lot of interesting aspects with that. s77 00:08:52.720 --> 00:08:55.440 Uh we described it in a recent blog post as well. s78 00:08:55.440 --> 00:08:57.920 You can find it on hornet.dev. s79 00:08:57.920 --> 00:09:02.800 And we like to compare it with a AOL query log. s80 00:09:02.800 --> 00:09:17.519 So AOL AOL was like a service back in the day had some search interface and they accidentally published um a very large sample of what people were searching for on the web s81 00:09:17.519 --> 00:09:28.080 and they were quite short and I have seen more recent query logs as well and the user human pattern are still searching with just a few terms. s82 00:09:28.080 --> 00:09:31.200 GPD5 on the other hand it's a much more powerful user. s83 00:09:31.200 --> 00:09:36.080 It has the general knowledge and it can like bam write out very long queries. s84 00:09:36.080 --> 00:09:42.480 It can use a lot of syntax operators that are kind of useful from it has learned from web search. s85 00:09:42.480 --> 00:09:49.440 So site operator phrases etc. And this is a new type of workload. s86 00:09:52.320 --> 00:10:02.160 And on BM25, BM25 has essentially two hyperparameters that controls various aspects of the scoring function. s87 00:10:02.160 --> 00:10:14.800 And I talked about having a baseline and BM25 is usually a baseline and browser comp plus as well has a baseline with BM25, but it turns out that that baseline is terrible. s88 00:10:14.800 --> 00:10:27.279 So when you look at fancier techniques, embedding models, what have you, um it's stands out as a much better retrieval paradigm than BM25 if you look at the original paper. s89 00:10:27.279 --> 00:10:37.600 But more recent research shows that the parameters that were used in the browse comp plus research paper was not really adequate to handle these kind of long documents. s90 00:10:37.600 --> 00:10:41.519 So I like this which BM25 do you mean? s91 00:10:41.519 --> 00:10:56.480 uh because it has a quite dramatic impact on that specific config benchmarks on the overall accuracy and why is BM25 now more powerful with the new user. s92 00:10:56.480 --> 00:11:09.519 So I mentioned the general knowledge of the user and that he can type faster and more be more specific as a more powerful user and exact matching is still relevant right because the model knows names entities s93 00:11:09.519 --> 00:11:21.440 uh zip codes uh skus what have you that is not so easy to represent with an embedding model which kind of encodes all the tokens into a fixed vocabulary. s94 00:11:21.440 --> 00:11:36.240 Uh it's also relatively cheap especially if you take into consideration the cost of doing embedding inference right some of these embedding models have like 8 billion parameters and you encode text and you have to stand up infrastructure for this and what have you. s95 00:11:36.240 --> 00:11:41.920 So it's quite simple and also the tooling uh in the overall ecosystem is quite good. s96 00:11:41.920 --> 00:11:54.480 Um, so it's it's readily available and it's also very easy for the model to inspect the results and understand why a certain query formulation returned the result it did, right? s97 00:11:54.480 --> 00:12:03.360 Because you're matching literal terms and phrases and things like that which can help it kind of reformulate the queries. s98 00:12:03.360 --> 00:12:10.880 So these are the three key things and now into more hot topics on like what is all you need. s99 00:12:10.880 --> 00:12:17.839 Um and this is a very recent research that came out from Waterloo from Jimmy Lynn's group up there. s100 00:12:17.839 --> 00:12:23.440 They are doing a great job at the information retrieval research and also on agentic search. s101 00:12:23.440 --> 00:12:26.240 They have a recent paper that I love. s102 00:12:26.240 --> 00:12:32.720 Uh it's called scaling direct corpus interaction via dynamic workspace expansion. s103 00:12:32.720 --> 00:12:35.040 So I'll spend some time on expanding this. s104 00:12:35.040 --> 00:12:39.680 So imagine you want to stand up web search infrastructure for agents. s105 00:12:39.680 --> 00:12:42.000 A lot of companies are doing that at the moment. s106 00:12:42.000 --> 00:12:48.880 We are also working with some of these companies to help them build infrastructure for powering that kind of use case. s107 00:12:48.880 --> 00:12:53.279 And there you have potentially billions of documents, right? s108 00:12:53.279 --> 00:12:55.440 So that doesn't fit into the context window. s109 00:12:55.440 --> 00:12:59.920 So you obviously need retrieval and BM25 is a good baseline. s110 00:12:59.920 --> 00:13:03.839 So you can retrieve information over that. s111 00:13:03.839 --> 00:13:21.680 And the result of this is um you can imagine this as a search engine result page SER for agents because you can place these documents that are retrieved from the retriever into a workspace. s112 00:13:21.680 --> 00:13:29.279 And if you organize this workspace as a file system, you can uh play into the same things that you have around skills. s113 00:13:29.279 --> 00:13:39.440 You can have progressive disclosure because you can have the document like the title of the document and a small snippet of the document and expose that to the model s114 00:13:39.440 --> 00:13:53.040 and the model can then also decide oh I need to read more of the document and when it's doing that it can use all the primitive tools that it's really good at using you all use coding agents so you all seen GP and rip gp and said and s115 00:13:53.040 --> 00:14:02.800 whatever it's doing to kind of manage context so here you get the benefits of both worlds and also So you get to combine sandbox infrastructure, retrieval infrastructure. s116 00:14:02.800 --> 00:14:05.519 So and VFS and just bash and whatever. s117 00:14:05.519 --> 00:14:07.199 So it's also quite exciting, right? s118 00:14:07.199 --> 00:14:11.279 Because it combines all of these new type of paradigms that is happening at the moment. s119 00:14:11.279 --> 00:14:14.240 So I'm really excited about this type of direction. s120 00:14:14.240 --> 00:14:20.000 And it's also kind of a hack, right, to optimize for what the models are good at at the moment, right? s121 00:14:20.000 --> 00:14:27.279 Because all the frontier LLM companies are optimizing their model for cool for coding, for bash, for tool use. s122 00:14:27.279 --> 00:14:37.279 So if you put your kind of end to end task on the trajectory of that you kind of whatever there's a new model you know there's going to be better at this as well right s123 00:14:37.279 --> 00:14:47.199 so might be that when we get AGI they can just use the browser we'll see but currently this is a very powerful way to to build uh retrieval infrastructure s124 00:14:47.199 --> 00:14:57.519 and a whole agentic search experience and when it comes to evaluation right I talked about this earlier as well. s125 00:14:57.519 --> 00:15:07.199 In traditional information retrieval, we were used to having just one query, one rank list, compute NDG and compare it. s126 00:15:07.199 --> 00:15:17.440 That is no longer very relevant when the new user is agent because that agent can reformulate queries and do more queries and do expansion and all kinds of different stuff, s127 00:15:17.440 --> 00:15:17.680 right? s128 00:15:17.680 --> 00:15:23.839 So, a lot of the classical information retrieval evaluation is now kind of dead. s129 00:15:23.839 --> 00:15:38.639 instead look at like see if the model can perform the task it's it's set to and for example for question answering does it get the answer right and we at hornet we are betting on BM25 s130 00:15:38.639 --> 00:15:51.920 as one of the primitives and we have set out on a vision to kind of have the best most efficient way to evaluate uh BM25 because I think it's such a strong fundamental s131 00:15:51.920 --> 00:16:08.079 primitive And this illustration compares some anonymized engines comparing with Hornet on the same type of same type of hardware with um web documents, 100 million web documents on a single node. s132 00:16:08.079 --> 00:16:22.959 And as you can see, Hornet has a much more efficient implementation than the other engines and can do more throughput for the same type of bucks, which for a lot of companies that are building infrastructure at the moment for web search etc. s133 00:16:22.959 --> 00:16:25.199 means a lot in savings. s134 00:16:25.199 --> 00:16:26.639 What is the yaxis? s135 00:16:26.639 --> 00:16:29.440 The y-axis is QPS. s136 00:16:29.440 --> 00:16:31.440 Sorry. s137 00:16:32.959 --> 00:16:34.639 The Y axis is Oh, sorry. s138 00:16:34.639 --> 00:16:36.320 The Y axis is latency. s139 00:16:36.320 --> 00:16:38.480 I'm sorry. s140 00:16:40.320 --> 00:16:44.320 So, four claims to take away from this talk. s141 00:16:44.320 --> 00:16:46.639 There's a new user. s142 00:16:46.639 --> 00:16:49.040 It's more powerful. s143 00:16:49.040 --> 00:16:51.279 It's able to type faster. s144 00:16:51.279 --> 00:16:52.959 It's able to read faster. s145 00:16:52.959 --> 00:17:03.040 It's able to reformulate queries and it has a lot of general knowledge which makes simple tools like GP and BM25 more powerful. s146 00:17:03.360 --> 00:17:04.559 That's number one. s147 00:17:04.559 --> 00:17:07.760 Number two, which BM25 do you mean? s148 00:17:07.760 --> 00:17:12.240 Uh there are differences in implementation, in performance, in parameters. s149 00:17:12.240 --> 00:17:14.559 So think about that. s150 00:17:14.559 --> 00:17:17.839 And also why is it effective for agentic search? s151 00:17:17.839 --> 00:17:20.559 It's simply because it's explainable for the model. s152 00:17:20.559 --> 00:17:24.480 So the model can see and also use it in combination with GP, right? s153 00:17:24.480 --> 00:17:29.120 Because you have literal matches and GP is almost about literal matches as well. s154 00:17:29.120 --> 00:17:37.919 And the combination of these two is a very strong agentic search or agentic retrieval paradigm. s155 00:17:39.600 --> 00:17:40.799 There's a lot of references. s156 00:17:40.799 --> 00:17:54.080 I think I will publish a talk uh or the talk will be published and also the the slides and if you hated it, you can tweet at me. s157 00:17:54.799 --> 00:18:01.600 Um there is not I was told that there's not a room for for questions but happy to chat about retrieval. s158 00:18:01.600 --> 00:18:03.679 You will find me around the conference. s159 00:18:03.679 --> 00:18:10.799 Probably best way to reach me is through my ex account and that's it.