{
  "video": {
    "id": "L4dUFimVDKU",
    "title": "Backtest Your Agent Against Yesterday's Web — Ryan Ahern, Exa",
    "duration": 1602,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": ""
  },
  "analysis": {
    "video_id": "L4dUFimVDKU",
    "title": "Backtest Your Agent Against Yesterday's Web — Ryan Ahern, Exa",
    "one_liner": "Exa Snapshot lets you run Exa's web search as it would have looked on a past date, so you can backtest agentic or search-driven predictive systems without future information leaking in. Use a holdout set, or people will overfit to the companies they can see.",
    "summary": "Ryan Ahern argues that AI agents now search far more than humans do, and that Exa supplies that search to products like Cursor and Gamma. He pitches Exa on quality, cost and latency. He then introduces Exa Snapshot, a versioned index of about 100 billion documents across trillions of URLs. It supports content versioning (a URL as of a date) and version search (running a query as of a past date), which lets you backtest systems without live search leaking the outcome. A hands-on competition had attendees write yes/no features to predict company acquisitions 1, 7 and 30 days ahead. The winner overfit by excluding the confounding companies by name, which showed why a holdout set matters.",
    "key_points": [
      "AI searches more than humans because it will run many searches and dig through multiple pages until it finds the answer. Humans type a few keywords and give up after page one.",
      "Exa provides search to coding agents and agentic apps such as Cursor (fetching docs) and Gamma (fetching facts like Bank of America's 2019 returns). Its finance customers include hedge funds, sovereign wealth funds, asset managers and VCs.",
      "Exa built its own 'regenerative benchmark' because a published static benchmark gets trained on and hill-climbed. The aim is benchmarks that users can build for their own use case and keep up to date.",
      "Cost: Exa tries to return only the tokens needed across all results, to cut input-token spend. Latency ranges from sub-300 ms searches to a 20-minute deep run that found 3,212 video game publishers and the developers they work with.",
      "Exa Snapshot stores a version each time Exa sees a page change or rediscovers it, across an index of about 100 billion documents tracking trillions of URLs. Content versioning returns a URL as of a date, or the nearest earlier version. Version search runs a whole query as of a past date.",
      "Live search leaks future knowledge into evaluations of the past. Setting the snapshot date just before an outcome, such as an acquisition announcement, ensures the confirmation isn't visible to the system.",
      "Article publish dates can't be trusted as features because publishers update them all the time, sometimes just because SEO likes fresh articles. Snapshot gives a reliable point-in-time view instead.",
      "Three backtesting patterns: (1) a traditional model that only uses Snapshot to collect historical features; (2) a traditional model plus agentic features (for example an adverse-media scan turned into a structured signal), tested and tuned one feature at a time; (3) fully agentic systems that choose when to search. One example is freezing the web just before a PR was merged in an external repo, to test a coding model without leakage.",
      "In the demo, 10 companies (some acquired, some confounders) were scored +1 for a correct acquisition call at 1, 7 or 30 days before, −1 for a false positive, and 0 for misses. Features could be combined with AND/OR. The winning entry used 'is this company publicly listed' plus 'is it a company other than Qiagen, Vitris, Bank of Cryptos, and American Electric Power'. That is blatant overfitting, made possible because the demo had no holdout set."
    ],
    "takeaways": [
      "When backtesting any system that uses web search, pin the search to the decision date (for example 1, 7 or 30 days before the outcome) rather than using live search, which leaks the answer.",
      "Add agentic web-search features to an existing model one at a time, and backtest each against known true and false outcomes to prove it adds value rather than relying on 'vibes'.",
      "Don't use article publish dates as trustworthy point-in-time features; use versioned snapshots of the content instead.",
      "Always keep a holdout set when iterating on features against a known set of outcomes. Otherwise the features will overfit to the visible examples."
    ],
    "topics": [
      "agentic search",
      "backtesting",
      "data leakage",
      "evals",
      "finance",
      "overfitting",
      "point-in-time data",
      "web search"
    ],
    "tools": [
      "Exa",
      "Exa Snapshot",
      "ChatGPT",
      "Claude",
      "OpenAI",
      "Cursor",
      "Gamma",
      "Axios",
      "The Verge"
    ],
    "quotes": [
      {
        "text": "Humans are kind of lazy. You'll be like, all right, I'm looking for this thing. I'll put in a few keywords. Hopefully it's on page one, otherwise I don't care.",
        "at": "02:01",
        "url": "https://www.youtube.com/watch?v=L4dUFimVDKU&t=121s"
      },
      {
        "text": "that also leaks kind of the future knowledge of that into an evaluation of the past.",
        "at": "10:27",
        "url": "https://www.youtube.com/watch?v=L4dUFimVDKU&t=627s"
      },
      {
        "text": "being able to version control it so I can prove that it adds value, not just like vibes and hope that it adds value.",
        "at": "17:11",
        "url": "https://www.youtube.com/watch?v=L4dUFimVDKU&t=1031s"
      },
      {
        "text": "You are the reason why I wanted to have a holdout",
        "at": "23:57",
        "url": "https://www.youtube.com/watch?v=L4dUFimVDKU&t=1437s"
      }
    ],
    "words": 6227
  },
  "summary_url": "/#L4dUFimVDKU",
  "transcript": {
    "html": "/transcripts/L4dUFimVDKU.html",
    "txt": "/transcripts/L4dUFimVDKU.txt",
    "vtt": "/transcripts/L4dUFimVDKU.vtt"
  }
}