Backtest Your Agent Against Yesterday's Web — Ryan Ahern, Exa

AI Engineer · 26 min · 351 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12Hello everyone.
00:12First of all, I'm Ryan.
00:13I'm a full stack engineer here at Exa.
00:15Very excited to walk you all through this very cool capability.
00:18Um we basically have perfect search for AI agents coming soon, but at least now you can version control it.
00:24Um what I'm going to take you through today is a few things.
00:27So, first off, just a little bit like what is Exa, what is agentic web search, why do I even care about this thing?
00:34At the end of point one, you're going to be like, I need agentic web search.
00:36I need it in my thing, regardless of you're doing anything financial or not.
00:40Number two, okay, I know I want this thing, but why do I care about version control?
00:44At the end of number two, you'll know why you care.
00:47And it'll be very important cuz on number three, there'll be a little hands-on exercise where you may or may not win a prize.
00:52Hopefully that motivates you to pay attention through points one and two.
00:55So, um I'm going to start off a little bit with like a show of hands just to situate.
00:59So, who here has heard of Exa before?
01:02Okay, we have a good few people, but a few people that haven't heard, which is perfect.
01:06How many of you have used like a search engine like a Google to do desktop research in a prior prior role?
01:12Like using Google just to search things around, at home, at work, whatever.
01:16Hopefully like most people, sweet.
01:18Then the last piece is going to be, hey, how many times have you now stopped using Google and instead use ChatGPT or Claude or some other agent and say, hey, can you find this thing out for me?
01:30Raise your hands if like now you do that.
01:32Sweet, great.
01:32Okay, perfect.
01:33We're going to tie all of those three things together because um the exact same thing that so many people just put their hands up, that's what we're seeing right now.
01:41So, AI is actually searching a lot more than humans do nowadays.
01:44Why?
01:44Cuz it can handle so much like back and forth, right?
01:47It can go out, it can do multiple searches, it can find what you're really looking for, can search through multiple pages, and work really hard to get you the answer you need.
01:55And we're starting seeing way more people are using AI to search and AI searches more and it searches more efficiently than humans.
02:01Humans are kind of lazy.
02:02You'll be like, all right, I'm looking for this thing.
02:04I'll put in a few keywords.
02:06Hopefully it's on page one, otherwise I don't care.
02:08Whereas AI systems, they really want to satisfy.
02:11These are the tend to be quite agreeable.
02:12I don't know if you found that out.
02:14And so they'll continue to searching until they find the thing.
02:16It's like all of these things mean there's a lot of search.
02:19And all of those tools, when you ask it a question, it's like, I need to look outside my knowledge window.
02:24I need to consult the internet.
02:26It will use a tool like Exa.
02:28And that's what we provide.
02:29So we actually provide search for some of the largest like coding agents and agentic applications in the world.
02:36So whenever you're using Cursor, and Cursor's like, gosh, if only I knew what the docs were, I could really help.
02:41It will use Exa to go out and find that doc and bring it back to its orchestration agent so it can answer the question.
02:47Or you're using Gamma and you're like, gee, if only I knew like what the financial returns were of like I don't know, Bank of America in 2019.
02:56This slide would pop.
02:57It's able to use Exa to go out and find that information.
02:59So that's kind of where we sit.
03:01I'm also just going to like do like a quick thing on finance cuz this is like Wall Street meets AI.
03:05So hopefully you're either in finance or you're interested in this.
03:08But we work across a lot of these areas.
03:10It's a particular area of interest to me.
03:12I work with some hedge funds and sovereign wealth funds and asset managers and VCs across these topics because they want information.
03:19Either they're like, you know, hedge fund and they're looking for alpha to to make financial decisions against.
03:25They're an asset manager, they're trying to forecast risk.
03:28They're a VC and they're like, oh, I want the hottest new startup in my pipeline and I got to find them before like The Verge or someone like in um um like puts up an article about it.
03:37They all leverage Exa.
03:39Um so hopefully by the end of this you're like, okay, yes.
03:44Um this is a really, really interesting space.
03:46But like kind of like why Exa?
03:48Like what do we do?
03:50Where we're a little bit different is that like we we of see ourselves as a lab and doing fundamental research to really push this forward.
03:57So, I'm going to touch on like the core triangle of like quality, cost, and latency really quick just so you know that you really want to put Exa in your application,
04:05and then we'll talk about versioning, um which would be like, "Ah, now I can actually see how and where this would be used to back test my application."
04:12So, first, quality.
04:14You want your results to be good.
04:15We care about that so much that we built our own regenerative benchmark.
04:19Why do you want to do that?
04:20Because search moves very quickly.
04:22You publish a benchmark and all of the search um providers and eventually the labs would be like, "We've got to train on that and climb the hill."
04:29So, what you really really care about, if you're a search engine that really believes in your stuff, is you want to create the uh you want to create the benchmarks that people can make for themselves against their use case and keep it up-to-date
04:41so that your search engine is working for you because you want to return the latest information.
04:44That's what the whole uh that's the whole reason why you're using a search provider and not just asking uh ChatGPT or OpenAI and answering from its weights.
04:52But, you're like, "Okay, I want quality, but I want it the lowest cost."
04:56We care about that not just at like the search time, but also, you know, search is a lot of tokens and tokens goes into a context window.
05:03And what does that mean?
05:04You're paying input costs.
05:05So, we developed technologies to try and return just the tokens that you need, not just across one result, but because all the results, so that your coding agent or whatever application can answer the question without having to pay a lot on input token costs.
05:18Lastly, latency.
05:20Really quickly, you can go very fast, so we have like sub-300 millisecond search all the way up to like, "Hey, work for like um I think I did one with like, 'Find me all the video game publishers' cuz I was trying to do some eval."
05:30It worked for 20 minutes and it found 3,212 of them and the developers that they work with.
05:35So, across the full spectrum, we've got you, no matter what point you're trying to implement it into your architecture.
05:40And now you're saying, "Ryan, okay, that's great.
05:42I really like Exa, but I want to version control it."
05:45Or, "Why do I want to version control it?
05:47Um and that's going to be the uh kind of focus of this discussion today to talk about this new capability we released called Exa Snapshot, which allows you to take our search as it is today,
05:58um but actually see what it would look like if you searched at a point in the past.
06:02I'm going to motivate like firstly what this is and then why you care about it.
06:06So, firstly, what is this?
06:07So, uh we're a search engine, so we spend a lot of time looking over the web.
06:12Uh and whenever we find this content, what we've been doing has been putting it into our index.
06:17And when we find like a new version of the page, or even when we just discover the same page at a date in the past, we store a version of that.
06:24Um and that means that like every URL across our full index, which is an index of the entire web with um everything around 100 billion documents tracking trillions of URLs,
06:34uh you can now go back in time.
06:37Um and there's two ways you can do that.
06:39The first one is like I know a URL, but I want to see what was that like in the past.
06:45Um and there's content versioning for that.
06:47So, you can go in, you can provide a date, and we will show you the content as of that date or the nearest date that we had before.
06:54So, you can guarantee that you have a version of that page that we would have served at that time.
06:59More broadly, you can use version search, which is not just like finding a URL, but executing a search as you would have done now, but in the past.
07:09Let's like dive into just like two very very small examples around this.
07:14Um and then like can kind of give like a vibe of some use cases.
07:18So, first one, let's say you're uh interested in financial markets.
07:23Who would be interested in financial financial markets around here?
07:26Um and one thing you might be trying to do is figure out like, "Hey, how can I understand the hottest and latest startups?"
07:31Maybe you guys would have heard of OpenAI.
07:32And there was some interesting news happening recently where um previously Axios and a few other people were reporting around uh OpenAI's uh annualized um like revenue and there's ideas like oh it might be up near like sort of 70 billion.
07:47And then more recently there's a lot of news about is it 70 billion or 50 billion?
07:51Turns out to be some like you know accounting management about whether it's gross revenue or net revenue, whether you're passing through certain cloud providers costs to you, do you recognize that revenue or not?
08:02But if you're a person who is in the business of understanding these things, it would be really helpful if you can be like hey I want to make sure
08:10that I'm designing a system that is able to look at these things, right?
08:15That is able to like at the point where you would have done that search, when you would have had that information, like is it tuned in such a way that it would reveal this like accounting footnote
08:26that could actually be very valuable to trade against.
08:30Let's say you're trying to do something arguably even more or less important, which is to understand the adverse media for like very important folks or maybe less important folks.
08:41And so like Ed Sheeran he had some like adverse media against him relatively recently.
08:46But if you look back a little bit before that event, you can see like hey what are a few other things that existed or you know he was he was not really good at insurance, maybe you wouldn't want to sponsor him for some insurance product.
08:57But the main thing here is that like at the point now where you're like hey I'm maybe I'm an insurance provider and I wanted to do some like counterparty due diligence or KYB or KYC
09:07or you're working like any business and you're trying to do partnerships and part of your flow is understanding another party.
09:15It would be good to be able to go back and be like hey I've designed this system to go out and do a bunch of web search and figure out what this person is, what this business is, how they could be classified,
09:24what whether we would insure against them or not.
09:27And now you're able to design that system and then with Hexo go back and test it.
09:31Um So that just gives like a few examples of sort of how and where uh, like snapshot can be useful in order to to kind of like backtest these systems.
09:42So, um, here's just like an an example of this.
09:45Um, as a note, this is important to take in some some awareness of because it may help with uh, you achieving some some some great excellence later.
09:56Um, but let's say you're like trying to understand something around like acquiring companies.
10:00So, you may be like, "Hey, I'm trying to design my system.
10:04If you do a search right now, I could see that this company is already acquired, but I might want to go back in the past and be like, "Hey, what about 1 day before the acquisition,
10:121 week before the acquisition, a month before the acquisition?
10:15Would I be able to find the information to predict that?"
10:18So, any live search necessarily is kept up to date.
10:22That's what you're kind of paying for, right?
10:24That if you did a search now, you'd have the latest information.
10:27But, um, that also leaks kind of the future knowledge of that into an evaluation of the past.
10:34If you're creating some predictive system, and all it does is goes like, "Okay, I look at the news and it says that this thing was acquired, so tick, it was acquired.
10:43100% successful true positive rate."
10:46It's assuming the search is good enough, which you hope it would be.
10:49Um, what's what's helpful with with snapshot and and kind of version version control search, you can constrain like what particular information is exposed at which time.
10:59So, here in this little example, I might set the snapshot date to like just before the outcome was announced.
11:05So, I know that the only thing that's going to be informing this system is uh, information that um, doesn't actually leak the exact confirmation of that thing, right?
11:16And then you could tune it to do different things.
11:18Like, maybe I have a feature that I really care about hiring.
11:21I want to make sure I find the right hiring signals.
11:24Maybe I have a feature that cares about like analyzing earning calls or or transcripts or or podcast interviews with the CEO.
11:31Whatever it is that's important to you, um, you're able to kind of tune that system and see, "Hey, would that have actually improved my prediction or would it not have improved my prediction?"
11:41I'm going to share just like and I'll spend like a tiny bit of time here but just talking about like a couple of patterns that we've seen in the backtesting space recently.
11:51So, the first one is around this idea of like a traditional model.
11:56So, the idea is here is you're just going to gather data that you need to backtest the model that you already have.
12:03You can kind of think of this is like, "Hey, maybe I have some system and it operates in the world.
12:08I don't need I don't need all that agentic web search.
12:10I love my system.
12:11It's amazing."
12:12Right?
12:13But maybe one thing that you can't do is actually go back and gather the features that you had in the past today.
12:20Right?
12:20Could be that they're based on web pages and those web pages just don't store a version that you can go back to.
12:27It could be that you're trying to have some sort of system that like takes like newspaper pages or some other articles and you can't really trust the publish date.
12:37Very small digression here but like publish dates of articles actually update all of the time.
12:42Both for like hey, like we've legitimately updated it all the way down to like hey, like SEO really likes up-to-date articles.
12:51So, maybe we just like update it one more time.
12:54So, that means that like you can't really trust that like publish date feature feature and is really important especially if you're making quite like substantial decisions around this that you're able to trust what that feature is.
13:04So, you may just decide, "Hey, I'm going to use snapshot.
13:08I know some URLs or some websites that I care about.
13:11I'm going to like jump back and just gather the data that I need and then put it in whatever feature I need, some array of numbers etc." X is not even involved in the prediction but this is like a perfectly valid and like relatively straightforward way that you might already be using some kinds of version data
13:27to backtest your system.
13:28Okay, that's the that's the existing space.
13:31Um what about the future?
13:33The future of perfect search.
13:34Okay, so there's two sort of patterns here.
13:36The first one is like I'm going to add an agentic feature to a traditional model.
13:42It's kind of similar to what we were talking about before, but the idea is in the live model, it's actually going to be using X or to like X to go out, search the web, turn all of that unstructured information
13:54into a structured signal, and then trade against it.
13:56So you could think about something like the adverse media scan like against our friend Ed Sheeran before.
14:03Um you you might be designing your system to be able to do some like counterparty due diligence, um and in the live production system, that might just be like one arm, right?
14:15It's uh the system itself may not have any agency, it's just going to collect a bunch of features, and then maybe you've like trained a model um um just like a traditional ML model to say like yes, we're going to invest, or no, we're not going to invest.
14:30You can uh go back and back test those agentic features one by one with X or and be like hey, uh against like a true positive match where something
14:40um like
14:40[clears throat]
14:41either like I don't know, the the the the thing that you care about happened, either like that business succeeded or it didn't succeed, you can test and be like were we able to correctly predict that,
14:50um or did we not?
14:52And most importantly, you can then tune that system.
14:54Because the way that with all of these non-deterministic systems, the way that you tune them is like really really important.
15:01So if you tune it to search a certain way, maybe it succeeds more often.
15:05If you choose to search another way, succeeds less often.
15:08And that is you can control that experiment and then add it into your system like one feature at a time.
15:13Um then like uh on the far would be right hand side, the idea is like there's this like future where there's some like predictions, startups, and other folks operating
15:23where potentially like the full system is agentic.
15:26Um so the system itself may or may not choose to search the web through some tool call.
15:32Um it could be uh combining internal knowledge.
15:35It could be combining like structured external knowledge.
15:38And it can kind of do that at run time.
15:40The idea is be like, "Hey, this is kind of what humans do when they go out and do some prediction.
15:44They have some agency.
15:45They're going to search for this or search for that."
15:47And essentially you can ground that uh like interface for your model into the outside world into the outside world.
15:56And then you can uh ensure that when that system was operating, it was able to search um only the information that it had at that time.
16:05So one like practical example of this is like let's say you're trying to improve your coding model.
16:10And hey, one part of that coding model would be uh whether it can solve a PR or not.
16:15You could be like, "Hey, I'm going to freeze right before that PR was solved uh in an in an external repo and then let the system run.
16:23And then hopefully, if it's able to resolve the task, you can have more confidence that that's a change in the model weights or whatever component in the system that you're actually paying to improve
16:32through your training process, then it is just like future information sort of like leaking into the current.
16:38Okay.
16:38So that's just like a little bit of like an overview of like snapshot and a few of the ways that it's being used to backtest.
16:45I'm just going to like uh recap what we've seen so far.
16:48So first, we really like Exa.
16:50Why do we like Exa?
16:51Because it's very good search.
16:53And search is very important for models because they need to know about uh the current uh and it needs to be high quality.
16:59It needs to be like uh cost effective.
17:01And it needs to be at the latency that makes sense for my system.
17:04Okay, we've all learned that.
17:05And then we're like, "Hey, I really want that.
17:08And a big part about putting that into my model is being able to version control it so I can prove that it adds value, not just like vibes and hope that it adds value.
17:17Um what we're going to do now is like the fun part of this workshop, which is we're going to give you all time to kind of go hands-on with this tool.
17:26So, there's a little QR code on the screen, you can scan it with a mobile.
17:29You can also just like enter this [clears throat] little URL into your computer and the uh interface on your laptop will probably be uh more amenable.
17:40Um and then I'm just going to give you like a little bit of an overview about what we're trying to do here.
17:44So, the idea is, if you think back to the example that we had before, um you're going to be trying to uh predict like what are company acquisitions right before they happen.
17:56Um to tell like a little bit of the story here.
17:58So, let's assume that you're working in some financial institution.
18:03Uh your boss's boss's boss's boss like went to some AI engineer workshop and they're like, "Hey, I heard this like really, really cool web search feature.
18:10I think we should present it to our portfolio manager."
18:13And portfolio manager's like, "Okay, I want to see it.
18:15You've got 20 minutes.
18:17Um try and see if you can make it good."
18:19Um and the most confounding variable here is you don't have like a coding agent.
18:23So, you have to do it by hand because your usage just ran out or something.
18:26Um so, that's the kind of context of where we're sitting ourselves at.
18:29Your task is going to be to find the best two features that like accurately predict a company acquisition.
18:37Uh and then Access Snapshot is going to use that to go back in time uh 1 day, 7 days, and 30 days before the acquisition and test your feature.
18:47Um or a random date if they weren't acquired, so there's a few confounding ones in there.
18:51Um feature is just like a yes/no question um that you believe would predict an acquisition.
18:57You can test as many as you like, there's a few examples in there, and I'm going to be coming around just to talk with you all um and kind of like help you along the way.
19:04Um and you can test as many as you like.
19:07Scoring, you get plus one point if you uh uh if your feature declares that this company would be an acquired uh on 170 30 days before it did.
19:18You get minus one if you say that it got acquired, uh but it didn't.
19:23And then, no impact for misses or passes cuz we're very friendly.
19:26Best score automatically captured, you can try as many as you want.
19:29I'm going to set us up for let's say like 30 minutes.
19:34Um so uh we'll kick that off now.
19:38And you have 30 minutes to try and put in the absolute best feature that you want.
19:42I'll be coming around.
19:43Maybe we can put on like a little bit of soft music that kind of gives people the vibe of working in
19:47[music]
19:47a portfolio manager company uh and listening to like low-fi beats.
19:51Uh and I'll come around and talk with you all.
19:55Okay, I'm just going to do like one like real quick example just to kind of show people how it works.
19:59If you're already going away and and and winning, keep feel free to keep grinding.
20:04Um but I'm just going to put one example in here.
20:05So, you'll see this interface in front of you.
20:07You have a bunch of um it'll ask you to put in a new feature.
20:10You can pick one of the examples or type whatever you want and hit run.
20:14Um and what it'll do is it'll go out, it'll use excess snapshot and search a bunch of them.
20:18This one was fast because it was cached, so uh that's a performance improvement to show you that we did test this beforehand.
20:25Um but it'll go down and it'll run that against this set of 10 companies.
20:29So, within this set of companies, some of them were acquired.
20:33Some of them were not acquired.
20:34They're confounding variables.
20:36Um and the idea is that like we're going to take this particular query, so has this company lost its CEO or CFO or appointed interim leadership.
20:44And for all of this, we're going to go back and do that version search.
20:46So, Delivery Hero, we can see, "Hey, um there was an executive leadership turnover.
20:51Yes."
20:52And the company did end up being acquired.
20:54So, hey, maybe this is predictive.
20:56Except, Quagen, I don't know if I'm pronouncing that correctly, they also had like executive leadership turnover, but they didn't end up being acquired.
21:03Oh, no, that's a confounding variable.
21:05So, your task is to try and be like, "Hey, what are the one or two features that I can add to this that will be the most predictive?"
21:12So, here I'm going to like add one more.
21:18So, this one is like, "Hey, has it been reported to be exploring a sale?"
21:21Does adding that improve it or make it worse?
21:24And in this case, you can see, "Hey, we've picked up a lot of the um uh we've picked up like a lot of the acquisitions, so we're doing pretty well in terms of finding stuff, but also like, "Hey, we're picking up like a lot of false alarms here."
21:35So, we might need to be more uh constraining in the way that we're combining these two features.
21:41So, here I've put a very simple like combined thing.
21:43You can either choose or, which is if either of your two features trigger yes, then we will count that whole thing as being like, "Yeah, you're predicting that it is going to be acquired
21:54as of that day."
21:55You can go and, and if I see if I go and here, the score improves a little bit, which means both of these two features need to be true.
22:01And if both of those two features are true for that company on that day, then you'll trigger yes.
22:06Um that's just like a PSA.
22:09I'll go back to the kind of like leadership board there, but feel free to chuck up your hands with any questions as we come through.
22:14Okay, 10 seconds left.
22:16If you haven't already hit run, it may be too late.
22:18Feel free to nobble over the combined and or to try and get your last few results.
22:24Phew.
22:24Okay, congratulations.
22:26So, Zane, John, and Ree, could you come and join me on stage if you won a prize?
22:30If not, we'll go further down in the list.
22:33Can we have a big round of applause?
22:40Come up, don't be shy.
22:41Don't be shy.
22:42Brilliant goods and mystery boxes await you.
22:47Hey, congratulations.
22:50Congratulations, John.
22:51Very valuable land.
22:56Um so, yeah, let me hand these out.
23:04Thank You're going to line up brilliantly so we can do a photo.
23:38Okay, we'll take a still out of that video later.
23:40So, let's just like poke around and be nosy and see like what did our number one person did.
23:44So, Zane Is this company a publicly listed company?
23:47Very interesting feature.
23:49Number two, is this company a company other than Qiagen, Vitris, Bank of Cryptos, and American Electric Power?
23:55Congratulations.
23:57You are the reason why I wanted to have a holdout
24:00[laughter]
24:02I removed it because I wanted it to be simple, but this is a good example of why you need to do backtesting and why you need to do backtesting with a holdout set.
24:09Um so, I'm going to take this as a feature, not a bug,
24:12[laughter]
24:13of not including a holdout set and why overfitting is very dangerous because I do not think that this particular algorithm would be successful in the future, but who knows?
24:22[laughter]
24:25Number two, Ree.
24:26So, okay, he stole Zane.
24:28[laughter]
24:29Maybe it would have been seated nearby.
24:30And lastly, with John.
24:31So, within the past 12 months, has an active investor or significant shareholder publicly called for a supported sale of companies as a whole or is company's current share price at least 60% below its high price during the past 5 years or is it named Dominion?
24:43Also, a little bit of overfitting there, but I would argue it probably performed better.
24:47Um so, this gives like an example of just like hey, how and why it's important to do backtesting, right?
24:52Because, you know, this is like a little constrained example.
24:55I I feel like I'm regretting removing my holdout set, although I do feel like people would have been way more confused if I did do that.
25:01Um but this gives you an idea of like why it's valuable, right?
25:04Because if you know what the existing set is, you can you can overfit it, but the idea is like because you have this capability, you can check for that, right?
25:12You can go back and be like, "Hey, I'm going to go back in my book and be like, hey, I'm going to constrain like these 10 companies that I know something had happened to them, 20 companies that didn't, and train and iterate on my model, right?"
25:23And make sure that it continues to perform beyond just like the publicly visible sets.
25:29Um so, let's like recap the highlights.
25:31I'm also going to Oh my god, there's a lot of people in here.
25:33I have to scroll very far.
25:34I'm going to put you all on a uh thank you page.
25:39I should already be on the thank you page.
25:40On the thank you page, you've got a link to our docs.
25:43We love documentation.
25:44You have a link to our blog.
25:46It's more readable than the docs
25:47[snorts]
25:47and is more like human-facing.
25:49Uh and lastly, there's a chance to send me an email.
25:52I really regret that now because there are 100 people in the room, so I'm expecting a lot of emails.
25:57Um but this is an enterprise feature as well.
25:59We have some public usage, but if you're interested in having access to this for your team, send me an email and I'll connect you to our enterprise team.
26:07Um let's just like recap the highlights.
26:08Number one, we love Exa.
26:10Why?
26:10Because it's great agentic web search.
26:13It's good value.
26:14It's very cost-effective.
26:15It's at whatever latency I need.
26:17I want it in my model.
26:18Number two, snapshot.
26:20It lets me backtest my model.
26:22It lets me see just how good Exa is at improving your model.
26:26And number three, um consider really adding the holdout set to your competition because otherwise you may overfit.
26:33Thank you, everyone.