{
  "video": {
    "id": "fZH97QHHYjY",
    "title": "The unreasonable effectiveness of BM25 for agentic search — Jo Kristian Bergum, Hornet.dev",
    "duration": 1109,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": "In person"
  },
  "analysis": {
    "video_id": "fZH97QHHYjY",
    "title": "The unreasonable effectiveness of BM25 for agentic search — Jo Kristian Bergum, Hornet.dev",
    "one_liner": "Jo Kristian Bergum argues BM25 is making a comeback not because the 30-year-old scoring function changed, but because the user changed: an LLM agent types faster, knows entities and syntax operators, and can read why a literal-match result came back and reformulate — making simple, cheap, explainable tools like BM25 and grep unreasonably effective inside agent loops.",
    "summary": "Bergum defines agentic search as search inside an agent loop, needing three things: a capable tool-using model, a harness (tool calling or 'code mode'), and a retrieval engine that can work over billion-scale corpora. He uses BrowseComp Plus (~830 riddle-like questions over ~100k web documents) to show that reasoning is not the bottleneck — stuffing the evidence documents directly into context gives high end-to-end accuracy even for GPT-4, while accuracy falls once the model has to retrieve for itself — so retrieval quality, not reasoning, caps task success. He then argues BM25's reputation as a weak baseline is partly a configuration artifact ('which BM25 do you mean?'), that its exact matching, low cost and explainability suit the new agent user, and that classical single-query IR metrics like NDCG are effectively dead for evaluating these systems.",
    "key_points": [
      "BM25 stands for 'best match 25' — experiment number 25 out of many turned out to be the best; the scoring function itself hasn't changed, the user has.",
      "Agentic search needs three components: a capable model that can use tools and formulate queries, a harness exposing retrieval (tool calling or 'code mode', as demoed by the prior speaker Edo), and a retrieval engine that can search efficiently over potentially billions of documents.",
      "On BrowseComp Plus (~830 pub-quiz-style questions, ~100–105k web documents, a single `search` tool returning snippets, golden reference answers), artificially stuffing the evidence documents into the context window yields very high accuracy even with GPT-4 — 'reasoning is not the bottleneck' — but accuracy drops once the model must retrieve, because it now depends on the harness, the model's query formulation and the retriever's quality.",
      "Context windows are 'one floppy disc' (1.4 MB) of data: in his opinion current models degrade past roughly 350,000 tokens, so even a perfect/AGI model still has to decide what goes into context — retrieval stays relevant.",
      "Analysis of GPT-5's BrowseComp Plus trajectories (written up in a Hornet.dev blog post) shows a new workload: where AOL-era and current human query logs are a few terms long, GPT-5 writes very long queries and uses web-search syntax operators like `site:` and phrase matching.",
      "The BM25 baseline in the BrowseComp Plus paper is 'terrible' because its two hyperparameters weren't adequate for long documents; more recent research shows configuration has a dramatic impact on overall accuracy — hence 'which BM25 do you mean?'",
      "BM25 fits agents because exact matching still matters for names, entities, zip codes and SKUs that embedding models struggle to represent in a fixed vocabulary; it's cheap compared to standing up inference for 8-billion-parameter embedding models; tooling is mature; and it's explainable, so the model can see why a query matched and reformulate.",
      "Jimmy Lin's Waterloo group paper 'scaling direct corpus interaction via dynamic workspace expansion' puts retrieved documents into a file-system workspace — a SERP for agents — with progressive disclosure (title + snippet, expand to read more) so the model can use grep, ripgrep, sed and bash; Bergum calls it a deliberate hack that rides the frontier labs' optimization for coding and tool use.",
      "Hornet's benchmark against anonymized engines on 100 million web documents on a single node, same hardware, shows higher throughput/lower latency per dollar (Bergum first said the y-axis was QPS, then corrected it to latency)."
    ],
    "takeaways": [
      "Stop treating BM25 as a throwaway baseline — tune its two hyperparameters for your document lengths before concluding embeddings beat it, and state which implementation and parameters you mean when you report numbers.",
      "Evaluate agentic retrieval end-to-end on task success (did the agent get the right answer?) rather than with single-query NDCG over one ranked list — the agent reformulates and issues many more queries than a human ever would.",
      "Diagnose your stack by first stuffing the gold evidence into context: if accuracy is high that way, your bottleneck is the harness and retriever, not the model's reasoning.",
      "Build the retrieval result as a file-system workspace with progressive disclosure so the agent can combine BM25 with the primitives it's already trained hard on — grep, ripgrep, sed, bash — instead of only returning snippets through a tool call.",
      "Prefer lexical exact matching for entities, names, zip codes and SKUs, and weigh BM25's cost against standing up and serving multi-billion-parameter embedding models."
    ],
    "topics": [
      "agentic search",
      "bm25",
      "information retrieval",
      "rag",
      "evals",
      "benchmarks",
      "context windows",
      "agent harnesses"
    ],
    "tools": [
      "Hornet.dev",
      "BM25",
      "BrowseComp Plus",
      "GPT-5",
      "GPT-4",
      "grep",
      "ripgrep",
      "sed",
      "bash",
      "AOL query log",
      "University of Waterloo / Jimmy Lin's group",
      "NDCG"
    ],
    "quotes": [
      {
        "text": "reasoning is not the bottleneck the model given the edit evidence up front answers the question with a very high accuracy rate even GPT4",
        "at": "07:23",
        "url": "https://www.youtube.com/watch?v=fZH97QHHYjY&t=443s"
      },
      {
        "text": "the current models before they start degrading in quality that's I in my opinion around 350,000 tokens. So that's one floppy disc of data right.",
        "at": "06:29",
        "url": "https://www.youtube.com/watch?v=fZH97QHHYjY&t=389s"
      },
      {
        "text": "a lot of the classical information retrieval evaluation is now kind of dead",
        "at": "15:18",
        "url": "https://www.youtube.com/watch?v=fZH97QHHYjY&t=918s"
      },
      {
        "text": "the combination of these two is a very strong agentic search or agentic retrieval paradigm",
        "at": "17:27",
        "url": "https://www.youtube.com/watch?v=fZH97QHHYjY&t=1047s"
      }
    ],
    "words": 3153
  },
  "summary_url": "/#fZH97QHHYjY",
  "transcript": {
    "html": "/transcripts/fZH97QHHYjY.html",
    "txt": "/transcripts/fZH97QHHYjY.txt",
    "vtt": "/transcripts/fZH97QHHYjY.vtt"
  }
}