{
  "video": {
    "id": "1IdzkRVmWAA",
    "title": "How we taught agents to use good retrieval - Hanna Lichtenberg, Mixedbread AI",
    "duration": 868,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": "Online"
  },
  "analysis": {
    "video_id": "1IdzkRVmWAA",
    "title": "How we taught agents to use good retrieval - Hanna Lichtenberg, Mixedbread AI",
    "one_liner": "Mixedbread argues the bottleneck in agentic knowledge work is not reasoning but retrieval — LLMs write grep-style keyword gibberish to search tools — and shows a purpose-built search agent (4 tools, max 4 parallel rounds, SFT + RL on a retrieval-plus-trajectory reward) that nearly closes the gap to Oracle performance.",
    "summary": "Amir and Hanna Lichtenberg of Mixedbread frame a 'knowledge gap': model reasoning has improved exponentially while search has crawled for 20 years, so agents with weak retrieval tools fall catastrophically short of Oracle performance (93% → 9 points on BrowseComp Plus, 64% → 8 points on Office QA Pro for Codex with its default tools). Swapping in Mixedbread search recovers nearly all of it — GPT-5 lands within 3 points of Oracle on BrowseComp Plus. They then show the queries models actually emit are keyword gibberish, because models are trained for grep-style code exploration, human-style web search, and BM25-favouring benchmarks like BEIR. Their fix is a harness plus a small trained agent: four differentiated search tools, goal framing, prompt tricks that forbid 'search query' phrasing, SFT from a teacher model, then on-policy RL with a combined retrieval and trajectory reward.",
    "key_points": [
      "Oracle performance (the model given the right documents) is 93% on BrowseComp Plus and 64% on Office QA Pro; Codex with its default tools scores 9 and 8 points respectively — the drop is retrieval, not reasoning.",
      "Dropping in Mixedbread search using 'latent interaction' recovers most of the gap: GPT-5 + Mixedbread is within 3 points of Oracle on BrowseComp Plus and almost fully closes it on Office QA Pro.",
      "BrowseComp Plus is OpenAI's BrowseComp fixed to a 100,000-document corpus instead of the open web; Office QA Pro is a Databricks benchmark over ~100 years of US treasuries.",
      "A real query found during benchmarking: 'senator woman questions billionaires not a company then okay thank you staff will check hearing' — models emit keyword soup because they are trained for grep/regex code exploration, trained to mimic human web-search patterns, and evaluated on caveman-style entity benchmarks (BEIR, NanoBEIR) that structurally favour BM25.",
      "The harness gives the agent four tools: overview search (up to 50 chunks, summaries only, to preview the corpus without filling context), semantic search (full payload, top 10 chunks), filter chunks (sort/filter on metadata facets), and grep for exact keyword matching.",
      "The loop is deliberately short for speed: max 4 search rounds, parallel searches within each round, the agent starts with an initial semantic search result plus available-metadata-facet hints, splits intent into at most 4 aspect-covering queries, dedupes returned chunks, and ends by submitting a ranked list of relevant chunks.",
      "Better queries come from five prompt design choices: making the agent state what evidence it needs first; genuinely distinct tools; instructing it to 'write one concise sentence describing what it wants to find' rather than 'write a search query'; few-shot examples of good queries and aspect splits; and showing corpus results up front so it can see the corpus language.",
      "Training is a small LLM (for speed) via supervised fine-tuning from a larger teacher, then on-policy RL with a reward combining retrieval (NDCG plus an LLM judge on rubrics: is the result relevant, are all chunks relevant, is the ranking plausible) and trajectory (LLM judge on whether queries are natural sentences and whether exploration is sufficient or excessive).",
      "Results: the trained agent is unreleased, but hits NDCG@10 of 0.4 on the Oblique Congress benchmark versus 0.18 for the best paper model (a GPT multi-hop agent); the production beta, Mixedbread agentic search, is top-1 on Snowflake's match QA benchmark at 93.4% accuracy with Gemini 3.5 Flash, at far less effort than comparable setups."
    ],
    "takeaways": [
      "Before blaming the model, measure the Oracle gap — run your task with the correct documents injected, and compare against what your current retrieval tools deliver; the difference is your retrieval bill, not a reasoning problem.",
      "Never instruct an agent to 'write a search query' against a semantic system — ask it to write one concise sentence describing what it wants to find, so it cannot fall back into BM25/grep keyword patterns.",
      "Give the agent differentiated retrieval tools rather than one search box: a wide summaries-only overview pass, a deep full-payload pass, metadata facet filtering, and grep reserved for genuine exact matching.",
      "Seed the agent's first turn with initial semantic search results and the available metadata facets so it can see the corpus vocabulary before planning, and dedupe chunks across rounds to protect context.",
      "If you train a search agent, reward the trajectory as well as the final ranking — judge whether queries are natural sentences and whether exploration was proportionate, not just NDCG."
    ],
    "topics": [
      "rag",
      "agents",
      "retrieval",
      "evals",
      "benchmarks",
      "reinforcement-learning",
      "prompt-engineering",
      "search"
    ],
    "tools": [
      "Mixedbread",
      "Mixedbread agentic search",
      "BrowseComp Plus",
      "BrowseComp",
      "OpenAI",
      "Office QA Pro",
      "Databricks",
      "Codex",
      "GPT-5",
      "Gemini 3.5 Flash",
      "Snowflake match QA benchmark",
      "Oblique Congress benchmark",
      "BEIR",
      "NanoBEIR",
      "BM25",
      "grep",
      "NDCG"
    ],
    "quotes": [
      {
        "text": "Meaning that actually the bottleneck here is not the reasoning. It's actually the access to the right knowledge it needs to answer this question.",
        "at": "03:05",
        "url": "https://www.youtube.com/watch?v=1IdzkRVmWAA&t=185s"
      },
      {
        "text": "It's basically gibberish, right? If you have a search system which wants to have neural questions or semantic question and just gets these keywords over keywords, it gets confused.",
        "at": "04:07",
        "url": "https://www.youtube.com/watch?v=1IdzkRVmWAA&t=247s"
      },
      {
        "text": "So, right now, the agent guesses the keywords to actually increase the overlap between the query and documents and can't really use powerful search tools properly.",
        "at": "05:11",
        "url": "https://www.youtube.com/watch?v=1IdzkRVmWAA&t=311s"
      },
      {
        "text": "We kind of trick the model into not thinking it has to write the typical BM25 base query by just instructing it to write one concise sentence describing what it wants to find, instead of directly instructing write a search query, so it cannot fall into this old pattern.",
        "at": "09:07",
        "url": "https://www.youtube.com/watch?v=1IdzkRVmWAA&t=547s"
      }
    ],
    "words": 2411
  },
  "summary_url": "/#1IdzkRVmWAA",
  "transcript": {
    "html": "/transcripts/1IdzkRVmWAA.html",
    "txt": "/transcripts/1IdzkRVmWAA.txt",
    "vtt": "/transcripts/1IdzkRVmWAA.vtt"
  }
}