Version-Controlled Search: The Web as It Was — Ryan Ahern, Exa

AI Engineer · 19 min · 207 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12Welcome everyone.
00:13Looking forward to giving you all a introduction into one of our newest features in Exa, which is version controlled search.
00:20So I'm Ryan, I'm a forward deployed engineer here at Exa.
00:23And today I'm going to give you an overview of two things.
00:26We only have a very short amount of time.
00:27So the first one, I just want to motivate what is Exa, what is agentic web search?
00:32Just because you kind of need to believe in the dream of that to understand why you'd want to use it to do the second thing I'm going to talk about, which is how to backtest financial models using that.
00:42So first, just to understand where I'm at in the room, who here has heard of Exa before or used it?
00:48Okay, I've got a good show of hands.
00:49Who here has ever used like Google back in the day to like do some sort of work or personal use, just like typed into Google to do some sort of search?
00:57Hopefully like everyone in the room, most people had pretty good market share, you might have heard of it.
01:01Last one would be, how many of you now no longer really use Google?
01:05You use ChatGPT or Claude or maybe even Instinct or Muse if you're like at the cutting edge to actually do your searches.
01:11Yeah, most people, right?
01:13So we're going to connect all of these things together.
01:16This last point where a lot of people now are like, "Hey, I don't even really use desktop search anymore.
01:22I kind of ask an agent to do this."
01:24is something that we see more broadly in the market, right?
01:26And the idea is that like agents tend to do a pretty good job at understanding how to search because they're allowed to try multiple, multiple times.
01:34They kind of don't get tired.
01:35They tend to be quite agreeable and work very, very hard.
01:38So when you say like this is the thing that I want, it's going to go out and it's going to do many, many different searches in order to satisfy
01:45like your objective.
01:46And the other thing here is that like they tend to search different to humans.
01:50So as a human, you might put in like a few keywords for the thing and then hope that you get the best results like in the first couple, and otherwise
01:57like maybe you try like a different query and then you'll kind of give up.
01:59Whereas agents will be very verbose.
02:02Um they they tend to describe what they want to search in great detail and will execute many searches in order to understand the space more fully before they respond.
02:10Um and that means that they do way more searches um on on a per task basis.
02:14So, generally the idea is that like, "Hey, nowadays most humans, you're going to be using agents to kind of broker your search tasks."
02:21Um and that's where Exa kind of plugs in.
02:23So, uh when you're working with um like many many agentic systems.
02:27So, whenever Cursor wants to go and search latest documentation to help its coding agent complete its trajectory, whenever you're like talking to Gamma and you're like, "Hey, it'd be great if I had just this like SEC data from like Apple in like 2018
02:41to complete my slide, like that would make it brilliant."
02:44It's going to use Exa to go out, search the full web, so we have our own crawling infrastructure, our own database, our own embedding model, our own retrieval architecture
02:52to get exactly what it needs to answer its query.
02:55Um this is like a couple of the logos of like the bigger companies, but because this is AI engineer, like here are a few like kind of archetypes that we work across the financial space.
03:04I work with a lot of hedge funds, asset managers, sovereign wealth funds, VCs across like a bunch of different tasks because they also need like the latest information.
03:13So, if you're a hedge funder, you're going to be trading on information.
03:16You're looking in the market for alpha, right?
03:18And for information that allows you to understand how you could position your your organization better.
03:24If you're a um uh an insurer, you're probably understanding like what counterparties you're working with or who you're insuring to make sure that ultimately you're making the determination.
03:34So, you want to make sure you're making it on the best information and that your risk models are the most accurate.
03:38Or even if you're in VC, you're like, "I need the hottest startup.
03:41I need to know the in next instant before they blow up."
03:44And they're not necessarily going to be in like some structured data set.
03:47Like I need to go and find them wherever they exist in the web.
03:50Um that's what they come to Exa to kind of support them for.
03:53Um and then the the kind of to wrap up the last part on like why Exa and the reason why I think now you'd be like, "Okay, I'm I'm building agents or I'm interested in agents and I definitely need a generative web search.
04:04Why Exa?"
04:06Um and it's because like we're a lab um in the sense that we're very very interested in the search market um and in the ways in which agents search and we're hyper focused on that problem.
04:15There's three like core tenants and then we'll talk about how that goes into versioning to segue into the next part of the talk.
04:20But first and quite obviously is like quality, right?
04:23So, you want to make sure that when the agent does that um the tool call and the retrieval, it gets what it needs cuz the fewer tool calls that it needs to make,
04:31the more cost-effective on your inference and also just like the more likely it is to succeed on its overall trajectory.
04:37You're not going to bring in something that like misaligns it.
04:39Um uh we're really really interested in the quality space.
04:43We uh recently released like a new benchmark um which we think more accurately assesses the um the the performance of like the search tools separately to the inference because these two things can be confounded, right?
04:56Like what agent or what model is calling the search and what search is connected to what model.
05:00So, that you can kind of have a look at that to see our latest thinking there.
05:03Secondly, on cost.
05:04So, it's not just the line item of the tool call, right?
05:07Like how much does one search cost because agents can broker many different searches.
05:11Um so, the idea is both like, "Hey, what's the value to this to complete your trajectory?"
05:15But also like what's the impact of this on the total cost of that agent completing its task cuz that's actually the thing that you care about, right?
05:23You're generally less opinionated about one tool call.
05:25It's like, "How much did this overall agent take?"
05:28Uh and that's why like we're we we worked on things like say dynamic highlights, which means that for every single result, we're not we're not just returning full pages.
05:36Uh we're not even just taking one page, which is our previous highlight features, and figuring out what are the exact right snippets on that page that we need to send back to the context window to complete the trajectory.
05:46We now do that across all of the search results, which means that you can actually get higher quality results at 95% fewer tokens than you would at full page text.
05:56We have a blog about this, it's very interesting, but basically it means you'll save a lot of money downstream on all of the input tokens you'll be spending on inference for the agent to complete its trajectory.
06:06Last one on latency, so because we own our own infrastructure, we can do very very low latency search.
06:12Um I'm just going to touch on the other side of the spectrum, which is like very long-running deep research.
06:17So we have ExaAgent Ultra, which we released um like not too long ago, maybe a few weeks ago, which is like state-of-the-art on um a lot of benchmarks around like very very long-running
06:28um search.
06:28And that's because we can do a lot of things under the hood.
06:31We have obviously we own the search harness, but we can also be opinionated about that how that interacts with the agent to harness do these like very very long-running research task.
06:39So highly recommend you check those things out.
06:41Um this is just a segway to be like, "Okay, Exa really cares about this space.
06:45I think that their search is going to be good."
06:47To motivate the next part, which is really interesting is the topic of this talk, which is versioning.
06:51So just recently we released ExaSnapshot.
06:54Uh it's 400 billion plus snapshots of the web, um and you can search them by date.
06:59Um and that's really really interesting in the um in the financial services industry.
07:05We'll get into that, but first I'm just going to talk a little bit about like what this basically means.
07:09So uh we're we're a search company, so we're constantly uh searching across the web for the latest information that we can add to our index.
07:17Um and every single time we do it, we check it for we're checking for freshness.
07:20Because ultimately you want to have the latest live up-to-date information at the point at which you're searching.
07:26But because we're doing all of those checks, it means that we can capture a version of those URLs at the point where we checked it and have like a version history.
07:34Um and you can access that in two ways.
07:35One is like, "I know the URL I want to find as it was at or before a given date."
07:40And so you can give us a URL and you say, "This is the date that I want it.
07:43I want it at that date or before and serve it to me as you would have if I had requested it at that point."
07:49The second one, which is even more exciting, is this concept of version search.
07:52So, where a search API, and the idea is like, "Hey, what would happen if I had executed that search but at a prior point, right?"
08:02So, I only get results that would have been served at that point or before.
08:06Um that has very like interesting implications.
08:09I'm just going to do one like very small example here.
08:12So, OpenAI recently I say as of like October 1st.
08:18So, on the left-hand side here, you'll see a snapshot search as it was in October 1st.
08:24And on the right-hand side was a live search as of like a couple of days ago when I made this slide.
08:28Um but uh at that time in early October, like there was some discussions that like OpenAI had a like annualized like run rate around like the 70 billion, right?
08:41Um and that was leading to certain ways in which the market was projecting how and where they would be valued.
08:46And then more recently, there was some and that reporting was originally by Axios.
08:49And then more recently, there were some publications that were reporting like, "Actually, it's closer to 50 billion for fun accounting reasons, right?"
08:56Like it turns out that they were counting gross revenue when actually like some of their revenue was coming from cloud providers.
09:03And technically, the way that you should account for that would be that there would be a vendor pass-through.
09:07So, net annualized revenue is better.
09:09That's how what some what some other players were reporting, etc., etc., right?
09:12But basically, it meant that the the the net annualized run rate was like kind of down forecast from 70 billion down to 50 billion, right?
09:21Now, this is just like a financial thing that some of you may or may not find interesting.
09:25But the so what for version search is like, "Let's say you're in the interest of creating a product or a solution or an agent that is made to that that will that its job is to to find these things, right?"
09:37You can now go back and be like, "Okay, before everyone figured it out, right?
09:41Before the market and before some financial analyst published the think piece that was like slamming down on this particular thing and everyone knew about it, what information existed on the internet, right?
09:51Like was there an SEC filing that would have revealed this thing in an accounting footnote?
09:55Like was there some other reporting from some other vendor that maybe is like you would consider is like less reputable, but actually was able to find that kind of thing and and maybe you should consider them differently."
10:05Um you can you can potentially like trace these things before like the rest of the market found it.
10:11But most importantly, like not only can you test out that you can backtest that, right?
10:15You could then adjust the ways in which you're setting up your model, you can adjust the ways that you're setting up your agent to be able to be like, "Could it have found this discrepancy?"
10:25Uh and more importantly, could it have found that discrepancy in a way that is generalizable, right?
10:30Let's like create a set of like like 50 sample sets, see if it can perform against that, check it against the holdout and be like, "Okay, I think that my model is now going to be pretty good at finding this kind of feature."
10:42And that's something that's kind of new in this in the in the web search space and that you can backtest that.
10:48Um this is like a like a little diagram just to kind of motivate the idea, but at any like live search that you do with like any existing search product, the idea is it's live, so it has all of the information.
11:00So, in this particular little example, um we're we're trying to predict whether company got acquired or not.
11:06Um in this case, if you do a live search, you're going to find like that the deal was announced, right?
11:10Because that happened, it got released everywhere.
11:13You could um be like, "Hey, I'm trying to train a model that's able to predict whether this company got acquired or not."
11:18So, I'm going to go like 1 day before the announcement, 7 days before the announcement, a month before the announcement and see whether I could or could not predict it, right?
11:26Uh and you can build a set of those companies um and basically train your model um or whatever system it is that it's trying to do this prediction, and see like was it able to reliably
11:36like predict this particular feature.
11:39Um So, I'm going to spend a little bit of time here just talking about how different systems are currently being back-tested with Snapshot.
11:49So, there's like three kind of archetypes here.
11:51The first one is I'm operating like a very traditional model.
11:56Maybe it's ML, maybe it's even more simple than that, like a decision tree or something.
12:00Uh and all I want to do is just gather unstructured data from the version web into some sort of structured set of information.
12:07So, it could be something like software prices, it could be some sentiment analysis, it could be like the terms and conditions of a particular site.
12:15Whatever it is that like powers your model, not all information is continually available.
12:20Like if you're trying to understand software prices, like, you know, the the page of that software price is not necessarily stored in a versioned um a way for that
12:31website in a way that you could reliably access.
12:35Or potentially like you have some like, sentiment analysis that you want to do, but like that's going to be mired in like what point was that sentiment released in time, right?
12:45And you want to be able to get like a reliable set of those signals.
12:48And then you can just transform it into some array of signals that goes into your model and you can predict it.
12:52That's like the sort of basic space.
12:55And some people are just like enriching data that they wouldn't have had access to for their existing models.
13:01So, that's just like sort of like the status quo plus one.
13:04The interesting space is in like the next two areas.
13:08So, the first one is this idea of like adding an agentic feature.
13:11And the And And here is like, okay, well, I have this existing model.
13:15It performs pretty well, but I have an idea that like if I add one or two like agentic features to this model, it would be able to perform better.
13:23So, let's say something like an adverse media scan.
13:26Necessarily like adverse media is going out into the web, searching across it for all of the information that exists, potentially doing it in an agentic way, right?
13:35Where you're doing multiple searches, you're exhausting multiple different trajectories, and then you're going to take all of that and then condense it into some sort of structure.
13:44Um Uh another example here would be to say like equity research in much the same way.
13:48Like, "Hey, I want to go out.
13:50I want to look at the full universe of companies and I want to find like potentially all of their like like estimated uh like rack revenue or head count or all of the features that matter."
14:00Um and the idea is that you'd be using something like adverse search at the point at which you're doing a prediction life to go out and gather all of that information
14:08to then send into your um traditional model.
14:11And you can now use like adverse snapshot to back test those particular features.
14:16So, you can go back and be like, "Hey, if I was to do this like adverse media scan like 1 month before."
14:21Or maybe I've already like onboarded a bunch of people into my portfolio like 4 months ago, what would I have said at the point where they were going to enter my portfolio?
14:30And would how is that tracked against today?
14:33Um and that kind of motivates like especially with a lot of financial services customers like you don't now just have to believe that like search will be better.
14:39You can actually go back and back test that against your existing processes and say, "Hey, it's better."
14:44Um so, that's like the maybe the next evolution like one step plus plus.
14:48Um on the far right-hand side here um is kind of like maybe the future.
14:53So, there's some like startups that are working in this prediction space.
14:57And the idea here is that the full sort of like simulation or prediction is being managed by an agent.
15:04So, in the middle is like an ML model that's maybe doing a prediction or some algorithm or decision tree.
15:08On the right-hand side like it is like an agent that's just going off and doing a bunch of things.
15:13It could be searching internal knowledge, it could be accessing data feeds, it could be doing whatever it is.
15:18But one of the things that it may do that that is to tool call uh and access the web.
15:24And the idea is like now you don't have to try and create some internal corpus that tries to represent what it would be like to access the full web.
15:33You can just be like, "Hey, I want to make sure that I can control that feature so I know that future information is not just going to leak directly into my model."
15:41Like I'm trying to predict company acquisitions through some very very insane like agentic process that's accessing everything.
15:47You can now be like, "All right, let's just make sure that's not just going to have one web search that finds that it was acquired, comes back and predicts it with like 100%
15:54accuracy.
15:55Um so that's kind of the the sort of right-hand side there.
15:59Um I'm just going to give like a little example of like the middle um just to round off um this this this kind of discussion.
16:06Um so uh yesterday we ran like um a kind of extended workshop on snapshot uh and we had a bunch of around about 100 people in the room that were kind of uh running through this the same example we had before about trying to predict company acquisitions.
16:20Um one thing that was kind of interesting is that um the this was like some like one of the example features from the workshop from one of the people
16:27um that uh that that got a relatively high score.
16:30But a thing that was interesting here is like um the just to just to motivate so the first one was this idea that like, "Hey, within the past year did an investor
16:39or a significant shareholder publicly call for a supported sale of the company as a whole?"
16:43And the second one, which is the interesting one, is like, "Hey, did this company's share price um was at least 60% lower than high share price during the past
16:515 years?"
16:51Which the idea is like it's kind of dipped down.
16:53Um the last bit that's interesting is like, "Hey, is the company named Dominion?"
16:58Now this is a little bit of overfitting for people that know like backtesting.
17:02Um like and the idea here is like this could be something that like an agent could do, right?
17:06If it's trained on like a live set, um it can potentially just overfit to the training data that you have.
17:11But the idea with snapshot is like not only can you go back and test for these things, um you can go back and like set up your your your e-vals and your identic harness in such a way that you can detect these things, but you can also try and check that it generalizes.
17:24And this is something that would be very, very hard to do if you are gated by like testing live, like in production.
17:30And you can only change the ways in which these features are described over time and check against real time, right?
17:36you might be waiting for months to try and see how and why your identic setup um, actually is predictive or is overfitting in different ways.
17:45Um, that is like you can describe these like quite complex features like um, these ones are like maybe you could imagine trying to recreate in um, uh, in like some sort of like uh, like non-semantic search, but you can go as crazy as you want within the within existing like
18:02uh, like extra semantic search world.
18:03So, you can say things like, "Hey, I want to see if there was any adverse media against like reputable VPs that are operating like um, uh, that are operating within VC funds that have like been on the AI Engineer podcast
18:16and they've said something bad about it, right?
18:17Like you can go as as hard as you want on these features that you think are going to be predictive cuz that's what core extra search allows, but then you can back test it, right?
18:25and see if it's performative or not.
18:27Um, then I'm just going to kind of like recap.
18:30So, the first thing that we talked about here is like um, using Exa because it's quality cost-effective identic web search across every latency.
18:38Um, we have lots of blogs around this, so encourage you if you've not tried Exa um, to have a go at this um, because the first thing you want to know is like, "Hey, do I actually think that this is going to add some benefit to my model?"
18:49The second thing as we talked about is like, "Hey, using Exa snapshot to be able to like back test these financial models or whichever model of risk risk model prediction model with these web search features
18:59in order to see like, hey, does adding it uh, add value to to you and your identic processes?"
19:04Thank you so much.