{
  "video": {
    "id": "T3SS931wU0I",
    "title": "Agentic Search vs Vector Search for Coding Agents: We Ran the Eval — Braintrust",
    "duration": 1095,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": ""
  },
  "analysis": {
    "video_id": "T3SS931wU0I",
    "title": "Agentic Search vs Vector Search for Coding Agents: We Ran the Eval — Braintrust",
    "one_liner": "On 20 bug-localization tasks from Microsoft's TypeScript Go repo, both run through the Claude Code harness, vector search and agentic search were equally accurate, but vector search cost about four times as much because it kept re-searching small chunks of code.",
    "summary": "Jess, a dev rel engineer at Braintrust, explains what evals are: a dataset, a task, a scoring system and experiments. She argues that teams should stop making ship decisions 'based off of vibes'. To show a full eval, she compares agentic search with vector search for a coding agent. Her CEO Ankur asked for it after seeing a Cursor tweet. She built the dataset from 'fix' PRs in Microsoft's TypeScript Go repo and ran both approaches through Claude Code. Accuracy was the same, vector search cost 4x more, and she openly lists the weaknesses of her own eval.",
    "key_points": [
      "Evals have four parts. The dataset holds golden-standard cases, edge cases and failure modes. The task is a system prompt plus the model that runs it. The scoring can be deterministic, LLM-as-a-judge or human review. Each combination of dataset, task and scorer is one experiment.",
      "She cites OpenAI's April 2025 model update, which was meant to make the model more helpful but made it sycophantic and less truthful, as the kind of regression evals should catch.",
      "Dataset: she found merged PRs titled 'fix' in Microsoft's open-source TypeScript Go repo and checked out each parent commit, so the bug was still in the code. Claude then wrote a bug or Linear-style task from the diff, and the agent had to find the buggy code. About 20 PRs were used.",
      "Agentic search was simply Claude Code's default behaviour. For vector search she still used the Claude Code harness, told it in the prompt not to use agentic search, and also used the disallowed tools flag, because 'prompting is never enough'.",
      "Claude Code ran as a subprocess, which left its traces orphaned. Passing the parent span IDs as environment variables made every turn, LLM call and grep/bash/ls/find command visible in the trace.",
      "Scoring was binary: 100% if the change passed the TypeScript Go test suite and 0% if it failed.",
      "Result: both approaches reached the same accuracy, and vector search cost about four times as much. Vector search returned chunks that left out imports, tool calls and the code calling them. In one case the vector agent made 26 searches and still couldn't work out how three functions in the file interacted.",
      "Agentic search worked like a human: it grepped the function name, read the whole function, spotted the flaw and followed the calling code into another file. Vector search got 'proximity' to the right code, while agentic search had the 'connective tissue'. The extra cost came from repeated search-and-return loops piling up LLM calls."
    ],
    "takeaways": [
      "Base ship decisions on eval numbers, such as '200 test cases, 94% passed' or '+5% accuracy but -5% tone', not on a few prompts that looked good.",
      "Build eval datasets from real data: sample 10–20 production logs, or mine merged 'fix' PRs and check out the parent commit to get real bug tasks.",
      "When you need to stop an agent harness from using some of its behaviours, block the tools with flags such as Claude Code's disallowed tools flag, not just prompt instructions.",
      "When an agent runs as a subprocess, pass the parent span IDs through environment variables so its traces stay connected and you can debug them.",
      "For more trustworthy results, run several trials per task, use a strong baseline implementation and expand the dataset beyond about 20 rows and one repo. Treat evals as a team sport involving AI engineers, PMs, subject matter experts and data analysts."
    ],
    "topics": [
      "evals",
      "agentic search",
      "vector search",
      "coding agents",
      "rag",
      "observability",
      "tracing",
      "llm cost"
    ],
    "tools": [
      "Braintrust",
      "Cursor",
      "Claude Code",
      "Claude",
      "OpenAI",
      "Microsoft TypeScript Go",
      "Qdrant",
      "Pinecone",
      "grep",
      "find",
      "ls",
      "cat",
      "bash",
      "Linear"
    ],
    "quotes": [
      {
        "text": "these teams are making ship decisions based off of vibes, uh which is not good.",
        "at": "01:04",
        "url": "https://www.youtube.com/watch?v=T3SS931wU0I&t=64s"
      },
      {
        "text": "But if you've ever done this before, you know that prompting is never enough.",
        "at": "12:59",
        "url": "https://www.youtube.com/watch?v=T3SS931wU0I&t=779s"
      },
      {
        "text": "vector search gave a lot of proximity to the correct code, but agentic search had more of the connective tissue to connect the actual logic of those chunks of code.",
        "at": "16:13",
        "url": "https://www.youtube.com/watch?v=T3SS931wU0I&t=973s"
      },
      {
        "text": "you might be sitting here thinking oh there's so many holes in this eval, which is very fair.",
        "at": "16:52",
        "url": "https://www.youtube.com/watch?v=T3SS931wU0I&t=1012s"
      }
    ],
    "words": 3758
  },
  "summary_url": "/#T3SS931wU0I",
  "transcript": {
    "html": "/transcripts/T3SS931wU0I.html",
    "txt": "/transcripts/T3SS931wU0I.txt",
    "vtt": "/transcripts/T3SS931wU0I.vtt"
  }
}