Version-Controlled Search: The Web as It Was — Ryan Ahern, Exa
https://www.youtube.com/watch?v=vS2IA9pq0Q4
AI Engineer · 19 min · 207 sentences · sentence-level transcript from YouTube's caption track; each line begins at the time shown

[00:01] [music]
[00:12] Welcome everyone.
[00:13] Looking forward to giving you all a introduction into one of our newest features in Exa, which is version controlled search.
[00:20] So I'm Ryan, I'm a forward deployed engineer here at Exa.
[00:23] And today I'm going to give you an overview of two things.
[00:26] We only have a very short amount of time.
[00:27] So the first one, I just want to motivate what is Exa, what is agentic web search?
[00:32] Just because you kind of need to believe in the dream of that to understand why you'd want to use it to do the second thing I'm going to talk about, which is how to backtest financial models using that.
[00:42] So first, just to understand where I'm at in the room, who here has heard of Exa before or used it?
[00:48] Okay, I've got a good show of hands.
[00:49] Who here has ever used like Google back in the day to like do some sort of work or personal use, just like typed into Google to do some sort of search?
[00:57] Hopefully like everyone in the room, most people had pretty good market share, you might have heard of it.
[01:01] Last one would be, how many of you now no longer really use Google?
[01:05] You use ChatGPT or Claude or maybe even Instinct or Muse if you're like at the cutting edge to actually do your searches.
[01:11] Yeah, most people, right?
[01:13] So we're going to connect all of these things together.
[01:16] This last point where a lot of people now are like, "Hey, I don't even really use desktop search anymore.
[01:22] I kind of ask an agent to do this."
[01:24] is something that we see more broadly in the market, right?
[01:26] And the idea is that like agents tend to do a pretty good job at understanding how to search because they're allowed to try multiple, multiple times.
[01:34] They kind of don't get tired.
[01:35] They tend to be quite agreeable and work very, very hard.
[01:38] So when you say like this is the thing that I want, it's going to go out and it's going to do many, many different searches in order to satisfy
[01:45] like your objective.
[01:46] And the other thing here is that like they tend to search different to humans.
[01:50] So as a human, you might put in like a few keywords for the thing and then hope that you get the best results like in the first couple, and otherwise
[01:57] like maybe you try like a different query and then you'll kind of give up.
[01:59] Whereas agents will be very verbose.
[02:02] Um they they tend to describe what they want to search in great detail and will execute many searches in order to understand the space more fully before they respond.
[02:10] Um and that means that they do way more searches um on on a per task basis.
[02:14] So, generally the idea is that like, "Hey, nowadays most humans, you're going to be using agents to kind of broker your search tasks."
[02:21] Um and that's where Exa kind of plugs in.
[02:23] So, uh when you're working with um like many many agentic systems.
[02:27] So, whenever Cursor wants to go and search latest documentation to help its coding agent complete its trajectory, whenever you're like talking to Gamma and you're like, "Hey, it'd be great if I had just this like SEC data from like Apple in like 2018
[02:41] to complete my slide, like that would make it brilliant."
[02:44] It's going to use Exa to go out, search the full web, so we have our own crawling infrastructure, our own database, our own embedding model, our own retrieval architecture
[02:52] to get exactly what it needs to answer its query.
[02:55] Um this is like a couple of the logos of like the bigger companies, but because this is AI engineer, like here are a few like kind of archetypes that we work across the financial space.
[03:04] I work with a lot of hedge funds, asset managers, sovereign wealth funds, VCs across like a bunch of different tasks because they also need like the latest information.
[03:13] So, if you're a hedge funder, you're going to be trading on information.
[03:16] You're looking in the market for alpha, right?
[03:18] And for information that allows you to understand how you could position your your organization better.
[03:24] If you're a um uh an insurer, you're probably understanding like what counterparties you're working with or who you're insuring to make sure that ultimately you're making the determination.
[03:34] So, you want to make sure you're making it on the best information and that your risk models are the most accurate.
[03:38] Or even if you're in VC, you're like, "I need the hottest startup.
[03:41] I need to know the in next instant before they blow up."
[03:44] And they're not necessarily going to be in like some structured data set.
[03:47] Like I need to go and find them wherever they exist in the web.
[03:50] Um that's what they come to Exa to kind of support them for.
[03:53] Um and then the the kind of to wrap up the last part on like why Exa and the reason why I think now you'd be like, "Okay, I'm I'm building agents or I'm interested in agents and I definitely need a generative web search.
[04:04] Why Exa?"
[04:06] Um and it's because like we're a lab um in the sense that we're very very interested in the search market um and in the ways in which agents search and we're hyper focused on that problem.
[04:15] There's three like core tenants and then we'll talk about how that goes into versioning to segue into the next part of the talk.
[04:20] But first and quite obviously is like quality, right?
[04:23] So, you want to make sure that when the agent does that um the tool call and the retrieval, it gets what it needs cuz the fewer tool calls that it needs to make,
[04:31] the more cost-effective on your inference and also just like the more likely it is to succeed on its overall trajectory.
[04:37] You're not going to bring in something that like misaligns it.
[04:39] Um uh we're really really interested in the quality space.
[04:43] We uh recently released like a new benchmark um which we think more accurately assesses the um the the performance of like the search tools separately to the inference because these two things can be confounded, right?
[04:56] Like what agent or what model is calling the search and what search is connected to what model.
[05:00] So, that you can kind of have a look at that to see our latest thinking there.
[05:03] Secondly, on cost.
[05:04] So, it's not just the line item of the tool call, right?
[05:07] Like how much does one search cost because agents can broker many different searches.
[05:11] Um so, the idea is both like, "Hey, what's the value to this to complete your trajectory?"
[05:15] But also like what's the impact of this on the total cost of that agent completing its task cuz that's actually the thing that you care about, right?
[05:23] You're generally less opinionated about one tool call.
[05:25] It's like, "How much did this overall agent take?"
[05:28] Uh and that's why like we're we we worked on things like say dynamic highlights, which means that for every single result, we're not we're not just returning full pages.
[05:36] Uh we're not even just taking one page, which is our previous highlight features, and figuring out what are the exact right snippets on that page that we need to send back to the context window to complete the trajectory.
[05:46] We now do that across all of the search results, which means that you can actually get higher quality results at 95% fewer tokens than you would at full page text.
[05:56] We have a blog about this, it's very interesting, but basically it means you'll save a lot of money downstream on all of the input tokens you'll be spending on inference for the agent to complete its trajectory.
[06:06] Last one on latency, so because we own our own infrastructure, we can do very very low latency search.
[06:12] Um I'm just going to touch on the other side of the spectrum, which is like very long-running deep research.
[06:17] So we have ExaAgent Ultra, which we released um like not too long ago, maybe a few weeks ago, which is like state-of-the-art on um a lot of benchmarks around like very very long-running
[06:28] um search.
[06:28] And that's because we can do a lot of things under the hood.
[06:31] We have obviously we own the search harness, but we can also be opinionated about that how that interacts with the agent to harness do these like very very long-running research task.
[06:39] So highly recommend you check those things out.
[06:41] Um this is just a segway to be like, "Okay, Exa really cares about this space.
[06:45] I think that their search is going to be good."
[06:47] To motivate the next part, which is really interesting is the topic of this talk, which is versioning.
[06:51] So just recently we released ExaSnapshot.
[06:54] Uh it's 400 billion plus snapshots of the web, um and you can search them by date.
[06:59] Um and that's really really interesting in the um in the financial services industry.
[07:05] We'll get into that, but first I'm just going to talk a little bit about like what this basically means.
[07:09] So uh we're we're a search company, so we're constantly uh searching across the web for the latest information that we can add to our index.
[07:17] Um and every single time we do it, we check it for we're checking for freshness.
[07:20] Because ultimately you want to have the latest live up-to-date information at the point at which you're searching.
[07:26] But because we're doing all of those checks, it means that we can capture a version of those URLs at the point where we checked it and have like a version history.
[07:34] Um and you can access that in two ways.
[07:35] One is like, "I know the URL I want to find as it was at or before a given date."
[07:40] And so you can give us a URL and you say, "This is the date that I want it.
[07:43] I want it at that date or before and serve it to me as you would have if I had requested it at that point."
[07:49] The second one, which is even more exciting, is this concept of version search.
[07:52] So, where a search API, and the idea is like, "Hey, what would happen if I had executed that search but at a prior point, right?"
[08:02] So, I only get results that would have been served at that point or before.
[08:06] Um that has very like interesting implications.
[08:09] I'm just going to do one like very small example here.
[08:12] So, OpenAI recently I say as of like October 1st.
[08:18] So, on the left-hand side here, you'll see a snapshot search as it was in October 1st.
[08:24] And on the right-hand side was a live search as of like a couple of days ago when I made this slide.
[08:28] Um but uh at that time in early October, like there was some discussions that like OpenAI had a like annualized like run rate around like the 70 billion, right?
[08:41] Um and that was leading to certain ways in which the market was projecting how and where they would be valued.
[08:46] And then more recently, there was some and that reporting was originally by Axios.
[08:49] And then more recently, there were some publications that were reporting like, "Actually, it's closer to 50 billion for fun accounting reasons, right?"
[08:56] Like it turns out that they were counting gross revenue when actually like some of their revenue was coming from cloud providers.
[09:03] And technically, the way that you should account for that would be that there would be a vendor pass-through.
[09:07] So, net annualized revenue is better.
[09:09] That's how what some what some other players were reporting, etc., etc., right?
[09:12] But basically, it meant that the the the net annualized run rate was like kind of down forecast from 70 billion down to 50 billion, right?
[09:21] Now, this is just like a financial thing that some of you may or may not find interesting.
[09:25] But the so what for version search is like, "Let's say you're in the interest of creating a product or a solution or an agent that is made to that that will that its job is to to find these things, right?"
[09:37] You can now go back and be like, "Okay, before everyone figured it out, right?
[09:41] Before the market and before some financial analyst published the think piece that was like slamming down on this particular thing and everyone knew about it, what information existed on the internet, right?
[09:51] Like was there an SEC filing that would have revealed this thing in an accounting footnote?
[09:55] Like was there some other reporting from some other vendor that maybe is like you would consider is like less reputable, but actually was able to find that kind of thing and and maybe you should consider them differently."
[10:05] Um you can you can potentially like trace these things before like the rest of the market found it.
[10:11] But most importantly, like not only can you test out that you can backtest that, right?
[10:15] You could then adjust the ways in which you're setting up your model, you can adjust the ways that you're setting up your agent to be able to be like, "Could it have found this discrepancy?"
[10:25] Uh and more importantly, could it have found that discrepancy in a way that is generalizable, right?
[10:30] Let's like create a set of like like 50 sample sets, see if it can perform against that, check it against the holdout and be like, "Okay, I think that my model is now going to be pretty good at finding this kind of feature."
[10:42] And that's something that's kind of new in this in the in the web search space and that you can backtest that.
[10:48] Um this is like a like a little diagram just to kind of motivate the idea, but at any like live search that you do with like any existing search product, the idea is it's live, so it has all of the information.
[11:00] So, in this particular little example, um we're we're trying to predict whether company got acquired or not.
[11:06] Um in this case, if you do a live search, you're going to find like that the deal was announced, right?
[11:10] Because that happened, it got released everywhere.
[11:13] You could um be like, "Hey, I'm trying to train a model that's able to predict whether this company got acquired or not."
[11:18] So, I'm going to go like 1 day before the announcement, 7 days before the announcement, a month before the announcement and see whether I could or could not predict it, right?
[11:26] Uh and you can build a set of those companies um and basically train your model um or whatever system it is that it's trying to do this prediction, and see like was it able to reliably
[11:36] like predict this particular feature.
[11:39] Um So, I'm going to spend a little bit of time here just talking about how different systems are currently being back-tested with Snapshot.
[11:49] So, there's like three kind of archetypes here.
[11:51] The first one is I'm operating like a very traditional model.
[11:56] Maybe it's ML, maybe it's even more simple than that, like a decision tree or something.
[12:00] Uh and all I want to do is just gather unstructured data from the version web into some sort of structured set of information.
[12:07] So, it could be something like software prices, it could be some sentiment analysis, it could be like the terms and conditions of a particular site.
[12:15] Whatever it is that like powers your model, not all information is continually available.
[12:20] Like if you're trying to understand software prices, like, you know, the the page of that software price is not necessarily stored in a versioned um a way for that
[12:31] website in a way that you could reliably access.
[12:35] Or potentially like you have some like, sentiment analysis that you want to do, but like that's going to be mired in like what point was that sentiment released in time, right?
[12:45] And you want to be able to get like a reliable set of those signals.
[12:48] And then you can just transform it into some array of signals that goes into your model and you can predict it.
[12:52] That's like the sort of basic space.
[12:55] And some people are just like enriching data that they wouldn't have had access to for their existing models.
[13:01] So, that's just like sort of like the status quo plus one.
[13:04] The interesting space is in like the next two areas.
[13:08] So, the first one is this idea of like adding an agentic feature.
[13:11] And the And And here is like, okay, well, I have this existing model.
[13:15] It performs pretty well, but I have an idea that like if I add one or two like agentic features to this model, it would be able to perform better.
[13:23] So, let's say something like an adverse media scan.
[13:26] Necessarily like adverse media is going out into the web, searching across it for all of the information that exists, potentially doing it in an agentic way, right?
[13:35] Where you're doing multiple searches, you're exhausting multiple different trajectories, and then you're going to take all of that and then condense it into some sort of structure.
[13:44] Um Uh another example here would be to say like equity research in much the same way.
[13:48] Like, "Hey, I want to go out.
[13:50] I want to look at the full universe of companies and I want to find like potentially all of their like like estimated uh like rack revenue or head count or all of the features that matter."
[14:00] Um and the idea is that you'd be using something like adverse search at the point at which you're doing a prediction life to go out and gather all of that information
[14:08] to then send into your um traditional model.
[14:11] And you can now use like adverse snapshot to back test those particular features.
[14:16] So, you can go back and be like, "Hey, if I was to do this like adverse media scan like 1 month before."
[14:21] Or maybe I've already like onboarded a bunch of people into my portfolio like 4 months ago, what would I have said at the point where they were going to enter my portfolio?
[14:30] And would how is that tracked against today?
[14:33] Um and that kind of motivates like especially with a lot of financial services customers like you don't now just have to believe that like search will be better.
[14:39] You can actually go back and back test that against your existing processes and say, "Hey, it's better."
[14:44] Um so, that's like the maybe the next evolution like one step plus plus.
[14:48] Um on the far right-hand side here um is kind of like maybe the future.
[14:53] So, there's some like startups that are working in this prediction space.
[14:57] And the idea here is that the full sort of like simulation or prediction is being managed by an agent.
[15:04] So, in the middle is like an ML model that's maybe doing a prediction or some algorithm or decision tree.
[15:08] On the right-hand side like it is like an agent that's just going off and doing a bunch of things.
[15:13] It could be searching internal knowledge, it could be accessing data feeds, it could be doing whatever it is.
[15:18] But one of the things that it may do that that is to tool call uh and access the web.
[15:24] And the idea is like now you don't have to try and create some internal corpus that tries to represent what it would be like to access the full web.
[15:33] You can just be like, "Hey, I want to make sure that I can control that feature so I know that future information is not just going to leak directly into my model."
[15:41] Like I'm trying to predict company acquisitions through some very very insane like agentic process that's accessing everything.
[15:47] You can now be like, "All right, let's just make sure that's not just going to have one web search that finds that it was acquired, comes back and predicts it with like 100%
[15:54] accuracy.
[15:55] Um so that's kind of the the sort of right-hand side there.
[15:59] Um I'm just going to give like a little example of like the middle um just to round off um this this this kind of discussion.
[16:06] Um so uh yesterday we ran like um a kind of extended workshop on snapshot uh and we had a bunch of around about 100 people in the room that were kind of uh running through this the same example we had before about trying to predict company acquisitions.
[16:20] Um one thing that was kind of interesting is that um the this was like some like one of the example features from the workshop from one of the people
[16:27] um that uh that that got a relatively high score.
[16:30] But a thing that was interesting here is like um the just to just to motivate so the first one was this idea that like, "Hey, within the past year did an investor
[16:39] or a significant shareholder publicly call for a supported sale of the company as a whole?"
[16:43] And the second one, which is the interesting one, is like, "Hey, did this company's share price um was at least 60% lower than high share price during the past
[16:51] 5 years?"
[16:51] Which the idea is like it's kind of dipped down.
[16:53] Um the last bit that's interesting is like, "Hey, is the company named Dominion?"
[16:58] Now this is a little bit of overfitting for people that know like backtesting.
[17:02] Um like and the idea here is like this could be something that like an agent could do, right?
[17:06] If it's trained on like a live set, um it can potentially just overfit to the training data that you have.
[17:11] But the idea with snapshot is like not only can you go back and test for these things, um you can go back and like set up your your your e-vals and your identic harness in such a way that you can detect these things, but you can also try and check that it generalizes.
[17:24] And this is something that would be very, very hard to do if you are gated by like testing live, like in production.
[17:30] And you can only change the ways in which these features are described over time and check against real time, right?
[17:36] you might be waiting for months to try and see how and why your identic setup um, actually is predictive or is overfitting in different ways.
[17:45] Um, that is like you can describe these like quite complex features like um, these ones are like maybe you could imagine trying to recreate in um, uh, in like some sort of like uh, like non-semantic search, but you can go as crazy as you want within the within existing like
[18:02] uh, like extra semantic search world.
[18:03] So, you can say things like, "Hey, I want to see if there was any adverse media against like reputable VPs that are operating like um, uh, that are operating within VC funds that have like been on the AI Engineer podcast
[18:16] and they've said something bad about it, right?
[18:17] Like you can go as as hard as you want on these features that you think are going to be predictive cuz that's what core extra search allows, but then you can back test it, right?
[18:25] and see if it's performative or not.
[18:27] Um, then I'm just going to kind of like recap.
[18:30] So, the first thing that we talked about here is like um, using Exa because it's quality cost-effective identic web search across every latency.
[18:38] Um, we have lots of blogs around this, so encourage you if you've not tried Exa um, to have a go at this um, because the first thing you want to know is like, "Hey, do I actually think that this is going to add some benefit to my model?"
[18:49] The second thing as we talked about is like, "Hey, using Exa snapshot to be able to like back test these financial models or whichever model of risk risk model prediction model with these web search features
[18:59] in order to see like, hey, does adding it uh, add value to to you and your identic processes?"
[19:04] Thank you so much.