WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=L4dUFimVDKU

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/L4dUFimVDKU.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.560 --> 00:00:12.960
Hello everyone.

s3
00:00:12.960 --> 00:00:13.800
First of all, I'm Ryan.

s4
00:00:13.800 --> 00:00:15.440
I'm a full stack engineer here at Exa.

s5
00:00:15.440 --> 00:00:18.880
Very excited to walk you all through this very cool capability.

s6
00:00:18.880 --> 00:00:24.400
Um we basically have perfect search for AI agents coming soon, but at least now you can version control it.

s7
00:00:24.400 --> 00:00:27.200
Um what I'm going to take you through today is a few things.

s8
00:00:27.200 --> 00:00:34.000
So, first off, just a little bit like what is Exa, what is agentic web search, why do I even care about this thing?

s9
00:00:34.000 --> 00:00:36.960
At the end of point one, you're going to be like, I need agentic web search.

s10
00:00:36.960 --> 00:00:40.920
I need it in my thing, regardless of you're doing anything financial or not.

s11
00:00:40.920 --> 00:00:44.760
Number two, okay, I know I want this thing, but why do I care about version control?

s12
00:00:44.760 --> 00:00:47.360
At the end of number two, you'll know why you care.

s13
00:00:47.360 --> 00:00:52.520
And it'll be very important cuz on number three, there'll be a little hands-on exercise where you may or may not win a prize.

s14
00:00:52.520 --> 00:00:55.640
Hopefully that motivates you to pay attention through points one and two.

s15
00:00:55.640 --> 00:00:59.480
So, um I'm going to start off a little bit with like a show of hands just to situate.

s16
00:00:59.480 --> 00:01:02.720
So, who here has heard of Exa before?

s17
00:01:02.720 --> 00:01:06.240
Okay, we have a good few people, but a few people that haven't heard, which is perfect.

s18
00:01:06.240 --> 00:01:12.880
How many of you have used like a search engine like a Google to do desktop research in a prior prior role?

s19
00:01:12.880 --> 00:01:16.480
Like using Google just to search things around, at home, at work, whatever.

s20
00:01:16.480 --> 00:01:18.280
Hopefully like most people, sweet.

s21
00:01:18.280 --> 00:01:30.160
Then the last piece is going to be, hey, how many times have you now stopped using Google and instead use ChatGPT or Claude or some other agent and say, hey, can you find this thing out for me?

s22
00:01:30.160 --> 00:01:32.040
Raise your hands if like now you do that.

s23
00:01:32.040 --> 00:01:32.680
Sweet, great.

s24
00:01:32.680 --> 00:01:33.440
Okay, perfect.

s25
00:01:33.440 --> 00:01:41.200
We're going to tie all of those three things together because um the exact same thing that so many people just put their hands up, that's what we're seeing right now.

s26
00:01:41.200 --> 00:01:44.560
So, AI is actually searching a lot more than humans do nowadays.

s27
00:01:44.560 --> 00:01:44.760
Why?

s28
00:01:44.760 --> 00:01:47.560
Cuz it can handle so much like back and forth, right?

s29
00:01:47.560 --> 00:01:55.120
It can go out, it can do multiple searches, it can find what you're really looking for, can search through multiple pages, and work really hard to get you the answer you need.

s30
00:01:55.120 --> 00:02:01.560
And we're starting seeing way more people are using AI to search and AI searches more and it searches more efficiently than humans.

s31
00:02:01.560 --> 00:02:02.840
Humans are kind of lazy.

s32
00:02:02.840 --> 00:02:04.680
You'll be like, all right, I'm looking for this thing.

s33
00:02:04.680 --> 00:02:06.320
I'll put in a few keywords.

s34
00:02:06.320 --> 00:02:08.679
Hopefully it's on page one, otherwise I don't care.

s35
00:02:08.679 --> 00:02:11.360
Whereas AI systems, they really want to satisfy.

s36
00:02:11.360 --> 00:02:12.840
These are the tend to be quite agreeable.

s37
00:02:12.840 --> 00:02:14.360
I don't know if you found that out.

s38
00:02:14.360 --> 00:02:16.400
And so they'll continue to searching until they find the thing.

s39
00:02:16.400 --> 00:02:19.480
It's like all of these things mean there's a lot of search.

s40
00:02:19.480 --> 00:02:24.720
And all of those tools, when you ask it a question, it's like, I need to look outside my knowledge window.

s41
00:02:24.720 --> 00:02:26.360
I need to consult the internet.

s42
00:02:26.360 --> 00:02:28.720
It will use a tool like Exa.

s43
00:02:28.720 --> 00:02:29.959
And that's what we provide.

s44
00:02:29.959 --> 00:02:36.160
So we actually provide search for some of the largest like coding agents and agentic applications in the world.

s45
00:02:36.160 --> 00:02:41.640
So whenever you're using Cursor, and Cursor's like, gosh, if only I knew what the docs were, I could really help.

s46
00:02:41.640 --> 00:02:47.800
It will use Exa to go out and find that doc and bring it back to its orchestration agent so it can answer the question.

s47
00:02:47.800 --> 00:02:56.080
Or you're using Gamma and you're like, gee, if only I knew like what the financial returns were of like I don't know, Bank of America in 2019.

s48
00:02:56.080 --> 00:02:57.320
This slide would pop.

s49
00:02:57.320 --> 00:02:59.920
It's able to use Exa to go out and find that information.

s50
00:02:59.920 --> 00:03:01.400
So that's kind of where we sit.

s51
00:03:01.400 --> 00:03:05.640
I'm also just going to like do like a quick thing on finance cuz this is like Wall Street meets AI.

s52
00:03:05.640 --> 00:03:08.880
So hopefully you're either in finance or you're interested in this.

s53
00:03:08.880 --> 00:03:10.760
But we work across a lot of these areas.

s54
00:03:10.760 --> 00:03:12.840
It's a particular area of interest to me.

s55
00:03:12.840 --> 00:03:19.440
I work with some hedge funds and sovereign wealth funds and asset managers and VCs across these topics because they want information.

s56
00:03:19.440 --> 00:03:25.400
Either they're like, you know, hedge fund and they're looking for alpha to to make financial decisions against.

s57
00:03:25.400 --> 00:03:28.000
They're an asset manager, they're trying to forecast risk.

s58
00:03:28.000 --> 00:03:37.640
They're a VC and they're like, oh, I want the hottest new startup in my pipeline and I got to find them before like The Verge or someone like in um um like puts up an article about it.

s59
00:03:37.640 --> 00:03:39.440
They all leverage Exa.

s60
00:03:39.440 --> 00:03:44.840
Um so hopefully by the end of this you're like, okay, yes.

s61
00:03:44.840 --> 00:03:46.720
Um this is a really, really interesting space.

s62
00:03:46.720 --> 00:03:48.560
But like kind of like why Exa?

s63
00:03:48.560 --> 00:03:50.840
Like what do we do?

s64
00:03:50.840 --> 00:03:57.840
Where we're a little bit different is that like we we of see ourselves as a lab and doing fundamental research to really push this forward.

s65
00:03:57.840 --> 00:04:05.360
So, I'm going to touch on like the core triangle of like quality, cost, and latency really quick just so you know that you really want to put Exa in your application,

s66
00:04:05.360 --> 00:04:12.600
and then we'll talk about versioning, um which would be like, "Ah, now I can actually see how and where this would be used to back test my application."

s67
00:04:12.600 --> 00:04:14.000
So, first, quality.

s68
00:04:14.000 --> 00:04:15.640
You want your results to be good.

s69
00:04:15.640 --> 00:04:19.000
We care about that so much that we built our own regenerative benchmark.

s70
00:04:19.000 --> 00:04:20.120
Why do you want to do that?

s71
00:04:20.120 --> 00:04:22.160
Because search moves very quickly.

s72
00:04:22.160 --> 00:04:29.400
You publish a benchmark and all of the search um providers and eventually the labs would be like, "We've got to train on that and climb the hill."

s73
00:04:29.400 --> 00:04:41.120
So, what you really really care about, if you're a search engine that really believes in your stuff, is you want to create the uh you want to create the benchmarks that people can make for themselves against their use case and keep it up-to-date

s74
00:04:41.120 --> 00:04:44.960
so that your search engine is working for you because you want to return the latest information.

s75
00:04:44.960 --> 00:04:52.919
That's what the whole uh that's the whole reason why you're using a search provider and not just asking uh ChatGPT or OpenAI and answering from its weights.

s76
00:04:52.919 --> 00:04:56.120
But, you're like, "Okay, I want quality, but I want it the lowest cost."

s77
00:04:56.120 --> 00:05:03.360
We care about that not just at like the search time, but also, you know, search is a lot of tokens and tokens goes into a context window.

s78
00:05:03.360 --> 00:05:04.200
And what does that mean?

s79
00:05:04.200 --> 00:05:05.640
You're paying input costs.

s80
00:05:05.640 --> 00:05:18.880
So, we developed technologies to try and return just the tokens that you need, not just across one result, but because all the results, so that your coding agent or whatever application can answer the question without having to pay a lot on input token costs.

s81
00:05:18.880 --> 00:05:20.360
Lastly, latency.

s82
00:05:20.360 --> 00:05:30.919
Really quickly, you can go very fast, so we have like sub-300 millisecond search all the way up to like, "Hey, work for like um I think I did one with like, 'Find me all the video game publishers' cuz I was trying to do some eval."

s83
00:05:30.919 --> 00:05:35.720
It worked for 20 minutes and it found 3,212 of them and the developers that they work with.

s84
00:05:35.720 --> 00:05:40.960
So, across the full spectrum, we've got you, no matter what point you're trying to implement it into your architecture.

s85
00:05:40.960 --> 00:05:42.680
And now you're saying, "Ryan, okay, that's great.

s86
00:05:42.680 --> 00:05:45.520
I really like Exa, but I want to version control it."

s87
00:05:45.520 --> 00:05:47.800
Or, "Why do I want to version control it?

s88
00:05:47.800 --> 00:05:58.040
Um and that's going to be the uh kind of focus of this discussion today to talk about this new capability we released called Exa Snapshot, which allows you to take our search as it is today,

s89
00:05:58.040 --> 00:06:02.000
um but actually see what it would look like if you searched at a point in the past.

s90
00:06:02.000 --> 00:06:06.560
I'm going to motivate like firstly what this is and then why you care about it.

s91
00:06:06.560 --> 00:06:07.880
So, firstly, what is this?

s92
00:06:07.880 --> 00:06:12.760
So, uh we're a search engine, so we spend a lot of time looking over the web.

s93
00:06:12.760 --> 00:06:17.240
Uh and whenever we find this content, what we've been doing has been putting it into our index.

s94
00:06:17.240 --> 00:06:24.520
And when we find like a new version of the page, or even when we just discover the same page at a date in the past, we store a version of that.

s95
00:06:24.520 --> 00:06:34.800
Um and that means that like every URL across our full index, which is an index of the entire web with um everything around 100 billion documents tracking trillions of URLs,

s96
00:06:34.800 --> 00:06:37.200
uh you can now go back in time.

s97
00:06:37.200 --> 00:06:39.080
Um and there's two ways you can do that.

s98
00:06:39.080 --> 00:06:45.040
The first one is like I know a URL, but I want to see what was that like in the past.

s99
00:06:45.040 --> 00:06:47.160
Um and there's content versioning for that.

s100
00:06:47.160 --> 00:06:54.120
So, you can go in, you can provide a date, and we will show you the content as of that date or the nearest date that we had before.

s101
00:06:54.120 --> 00:06:59.840
So, you can guarantee that you have a version of that page that we would have served at that time.

s102
00:06:59.840 --> 00:07:09.640
More broadly, you can use version search, which is not just like finding a URL, but executing a search as you would have done now, but in the past.

s103
00:07:09.640 --> 00:07:14.360
Let's like dive into just like two very very small examples around this.

s104
00:07:14.360 --> 00:07:18.640
Um and then like can kind of give like a vibe of some use cases.

s105
00:07:18.640 --> 00:07:23.440
So, first one, let's say you're uh interested in financial markets.

s106
00:07:23.440 --> 00:07:26.160
Who would be interested in financial financial markets around here?

s107
00:07:26.160 --> 00:07:31.000
Um and one thing you might be trying to do is figure out like, "Hey, how can I understand the hottest and latest startups?"

s108
00:07:31.000 --> 00:07:32.919
Maybe you guys would have heard of OpenAI.

s109
00:07:32.919 --> 00:07:47.680
And there was some interesting news happening recently where um previously Axios and a few other people were reporting around uh OpenAI's uh annualized um like revenue and there's ideas like oh it might be up near like sort of 70 billion.

s110
00:07:47.680 --> 00:07:51.800
And then more recently there's a lot of news about is it 70 billion or 50 billion?

s111
00:07:51.800 --> 00:08:02.240
Turns out to be some like you know accounting management about whether it's gross revenue or net revenue, whether you're passing through certain cloud providers costs to you, do you recognize that revenue or not?

s112
00:08:02.240 --> 00:08:10.000
But if you're a person who is in the business of understanding these things, it would be really helpful if you can be like hey I want to make sure

s113
00:08:10.000 --> 00:08:15.560
that I'm designing a system that is able to look at these things, right?

s114
00:08:15.560 --> 00:08:26.320
That is able to like at the point where you would have done that search, when you would have had that information, like is it tuned in such a way that it would reveal this like accounting footnote

s115
00:08:26.320 --> 00:08:30.600
that could actually be very valuable to trade against.

s116
00:08:30.600 --> 00:08:41.200
Let's say you're trying to do something arguably even more or less important, which is to understand the adverse media for like very important folks or maybe less important folks.

s117
00:08:41.200 --> 00:08:46.640
And so like Ed Sheeran he had some like adverse media against him relatively recently.

s118
00:08:46.640 --> 00:08:57.200
But if you look back a little bit before that event, you can see like hey what are a few other things that existed or you know he was he was not really good at insurance, maybe you wouldn't want to sponsor him for some insurance product.

s119
00:08:57.200 --> 00:09:07.960
But the main thing here is that like at the point now where you're like hey I'm maybe I'm an insurance provider and I wanted to do some like counterparty due diligence or KYB or KYC

s120
00:09:07.960 --> 00:09:15.160
or you're working like any business and you're trying to do partnerships and part of your flow is understanding another party.

s121
00:09:15.160 --> 00:09:24.839
It would be good to be able to go back and be like hey I've designed this system to go out and do a bunch of web search and figure out what this person is, what this business is, how they could be classified,

s122
00:09:24.839 --> 00:09:27.240
what whether we would insure against them or not.

s123
00:09:27.240 --> 00:09:31.920
And now you're able to design that system and then with Hexo go back and test it.

s124
00:09:31.920 --> 00:09:42.280
Um So that just gives like a few examples of sort of how and where uh, like snapshot can be useful in order to to kind of like backtest these systems.

s125
00:09:42.280 --> 00:09:45.640
So, um, here's just like an an example of this.

s126
00:09:45.640 --> 00:09:56.080
Um, as a note, this is important to take in some some awareness of because it may help with uh, you achieving some some some great excellence later.

s127
00:09:56.080 --> 00:10:00.640
Um, but let's say you're like trying to understand something around like acquiring companies.

s128
00:10:00.640 --> 00:10:04.040
So, you may be like, "Hey, I'm trying to design my system.

s129
00:10:04.040 --> 00:10:12.600
If you do a search right now, I could see that this company is already acquired, but I might want to go back in the past and be like, "Hey, what about 1 day before the acquisition,

s130
00:10:12.600 --> 00:10:15.880
1 week before the acquisition, a month before the acquisition?

s131
00:10:15.880 --> 00:10:18.760
Would I be able to find the information to predict that?"

s132
00:10:18.760 --> 00:10:22.880
So, any live search necessarily is kept up to date.

s133
00:10:22.880 --> 00:10:24.520
That's what you're kind of paying for, right?

s134
00:10:24.520 --> 00:10:27.840
That if you did a search now, you'd have the latest information.

s135
00:10:27.840 --> 00:10:34.480
But, um, that also leaks kind of the future knowledge of that into an evaluation of the past.

s136
00:10:34.480 --> 00:10:43.320
If you're creating some predictive system, and all it does is goes like, "Okay, I look at the news and it says that this thing was acquired, so tick, it was acquired.

s137
00:10:43.320 --> 00:10:46.280
100% successful true positive rate."

s138
00:10:46.280 --> 00:10:49.480
It's assuming the search is good enough, which you hope it would be.

s139
00:10:49.480 --> 00:10:59.000
Um, what's what's helpful with with snapshot and and kind of version version control search, you can constrain like what particular information is exposed at which time.

s140
00:10:59.000 --> 00:11:05.440
So, here in this little example, I might set the snapshot date to like just before the outcome was announced.

s141
00:11:05.440 --> 00:11:16.760
So, I know that the only thing that's going to be informing this system is uh, information that um, doesn't actually leak the exact confirmation of that thing, right?

s142
00:11:16.760 --> 00:11:18.600
And then you could tune it to do different things.

s143
00:11:18.600 --> 00:11:21.920
Like, maybe I have a feature that I really care about hiring.

s144
00:11:21.920 --> 00:11:24.080
I want to make sure I find the right hiring signals.

s145
00:11:24.080 --> 00:11:31.120
Maybe I have a feature that cares about like analyzing earning calls or or transcripts or or podcast interviews with the CEO.

s146
00:11:31.120 --> 00:11:41.200
Whatever it is that's important to you, um, you're able to kind of tune that system and see, "Hey, would that have actually improved my prediction or would it not have improved my prediction?"

s147
00:11:41.520 --> 00:11:51.560
I'm going to share just like and I'll spend like a tiny bit of time here but just talking about like a couple of patterns that we've seen in the backtesting space recently.

s148
00:11:51.560 --> 00:11:56.960
So, the first one is around this idea of like a traditional model.

s149
00:11:56.960 --> 00:12:03.400
So, the idea is here is you're just going to gather data that you need to backtest the model that you already have.

s150
00:12:03.400 --> 00:12:08.000
You can kind of think of this is like, "Hey, maybe I have some system and it operates in the world.

s151
00:12:08.000 --> 00:12:10.840
I don't need I don't need all that agentic web search.

s152
00:12:10.840 --> 00:12:11.640
I love my system.

s153
00:12:11.640 --> 00:12:12.560
It's amazing."

s154
00:12:12.560 --> 00:12:13.040
Right?

s155
00:12:13.040 --> 00:12:20.080
But maybe one thing that you can't do is actually go back and gather the features that you had in the past today.

s156
00:12:20.080 --> 00:12:20.560
Right?

s157
00:12:20.560 --> 00:12:27.160
Could be that they're based on web pages and those web pages just don't store a version that you can go back to.

s158
00:12:27.160 --> 00:12:37.800
It could be that you're trying to have some sort of system that like takes like newspaper pages or some other articles and you can't really trust the publish date.

s159
00:12:37.800 --> 00:12:42.800
Very small digression here but like publish dates of articles actually update all of the time.

s160
00:12:42.800 --> 00:12:51.520
Both for like hey, like we've legitimately updated it all the way down to like hey, like SEO really likes up-to-date articles.

s161
00:12:51.520 --> 00:12:54.200
So, maybe we just like update it one more time.

s162
00:12:54.200 --> 00:13:04.640
So, that means that like you can't really trust that like publish date feature feature and is really important especially if you're making quite like substantial decisions around this that you're able to trust what that feature is.

s163
00:13:04.640 --> 00:13:08.360
So, you may just decide, "Hey, I'm going to use snapshot.

s164
00:13:08.360 --> 00:13:11.280
I know some URLs or some websites that I care about.

s165
00:13:11.280 --> 00:13:27.120
I'm going to like jump back and just gather the data that I need and then put it in whatever feature I need, some array of numbers etc." X is not even involved in the prediction but this is like a perfectly valid and like relatively straightforward way that you might already be using some kinds of version data

s166
00:13:27.120 --> 00:13:28.560
to backtest your system.

s167
00:13:28.560 --> 00:13:31.440
Okay, that's the that's the existing space.

s168
00:13:31.440 --> 00:13:33.000
Um what about the future?

s169
00:13:33.000 --> 00:13:34.200
The future of perfect search.

s170
00:13:34.200 --> 00:13:36.800
Okay, so there's two sort of patterns here.

s171
00:13:36.800 --> 00:13:42.280
The first one is like I'm going to add an agentic feature to a traditional model.

s172
00:13:42.280 --> 00:13:54.080
It's kind of similar to what we were talking about before, but the idea is in the live model, it's actually going to be using X or to like X to go out, search the web, turn all of that unstructured information

s173
00:13:54.080 --> 00:13:56.880
into a structured signal, and then trade against it.

s174
00:13:56.880 --> 00:14:03.280
So you could think about something like the adverse media scan like against our friend Ed Sheeran before.

s175
00:14:03.280 --> 00:14:15.160
Um you you might be designing your system to be able to do some like counterparty due diligence, um and in the live production system, that might just be like one arm, right?

s176
00:14:15.160 --> 00:14:30.120
It's uh the system itself may not have any agency, it's just going to collect a bunch of features, and then maybe you've like trained a model um um just like a traditional ML model to say like yes, we're going to invest, or no, we're not going to invest.

s177
00:14:30.120 --> 00:14:40.200
You can uh go back and back test those agentic features one by one with X or and be like hey, uh against like a true positive match where something

s178
00:14:40.200 --> 00:14:40.985
um like

s179
00:14:40.985 --> 00:14:41.280
[clears throat]

s180
00:14:41.280 --> 00:14:50.760
either like I don't know, the the the the thing that you care about happened, either like that business succeeded or it didn't succeed, you can test and be like were we able to correctly predict that,

s181
00:14:50.760 --> 00:14:52.120
um or did we not?

s182
00:14:52.120 --> 00:14:54.760
And most importantly, you can then tune that system.

s183
00:14:54.760 --> 00:15:01.000
Because the way that with all of these non-deterministic systems, the way that you tune them is like really really important.

s184
00:15:01.000 --> 00:15:05.360
So if you tune it to search a certain way, maybe it succeeds more often.

s185
00:15:05.360 --> 00:15:08.320
If you choose to search another way, succeeds less often.

s186
00:15:08.320 --> 00:15:13.320
And that is you can control that experiment and then add it into your system like one feature at a time.

s187
00:15:13.320 --> 00:15:23.240
Um then like uh on the far would be right hand side, the idea is like there's this like future where there's some like predictions, startups, and other folks operating

s188
00:15:23.240 --> 00:15:26.360
where potentially like the full system is agentic.

s189
00:15:26.360 --> 00:15:32.000
Um so the system itself may or may not choose to search the web through some tool call.

s190
00:15:32.000 --> 00:15:35.520
Um it could be uh combining internal knowledge.

s191
00:15:35.520 --> 00:15:38.280
It could be combining like structured external knowledge.

s192
00:15:38.280 --> 00:15:40.280
And it can kind of do that at run time.

s193
00:15:40.280 --> 00:15:44.560
The idea is be like, "Hey, this is kind of what humans do when they go out and do some prediction.

s194
00:15:44.560 --> 00:15:45.760
They have some agency.

s195
00:15:45.760 --> 00:15:47.640
They're going to search for this or search for that."

s196
00:15:47.640 --> 00:15:56.320
And essentially you can ground that uh like interface for your model into the outside world into the outside world.

s197
00:15:56.320 --> 00:16:05.640
And then you can uh ensure that when that system was operating, it was able to search um only the information that it had at that time.

s198
00:16:05.640 --> 00:16:10.560
So one like practical example of this is like let's say you're trying to improve your coding model.

s199
00:16:10.560 --> 00:16:15.680
And hey, one part of that coding model would be uh whether it can solve a PR or not.

s200
00:16:15.680 --> 00:16:23.440
You could be like, "Hey, I'm going to freeze right before that PR was solved uh in an in an external repo and then let the system run.

s201
00:16:23.440 --> 00:16:32.400
And then hopefully, if it's able to resolve the task, you can have more confidence that that's a change in the model weights or whatever component in the system that you're actually paying to improve

s202
00:16:32.400 --> 00:16:38.000
through your training process, then it is just like future information sort of like leaking into the current.

s203
00:16:38.000 --> 00:16:38.680
Okay.

s204
00:16:38.680 --> 00:16:45.480
So that's just like a little bit of like an overview of like snapshot and a few of the ways that it's being used to backtest.

s205
00:16:45.480 --> 00:16:48.240
I'm just going to like uh recap what we've seen so far.

s206
00:16:48.240 --> 00:16:50.440
So first, we really like Exa.

s207
00:16:50.440 --> 00:16:51.640
Why do we like Exa?

s208
00:16:51.640 --> 00:16:53.000
Because it's very good search.

s209
00:16:53.000 --> 00:16:59.280
And search is very important for models because they need to know about uh the current uh and it needs to be high quality.

s210
00:16:59.280 --> 00:17:01.680
It needs to be like uh cost effective.

s211
00:17:01.680 --> 00:17:04.560
And it needs to be at the latency that makes sense for my system.

s212
00:17:04.560 --> 00:17:05.760
Okay, we've all learned that.

s213
00:17:05.760 --> 00:17:08.720
And then we're like, "Hey, I really want that.

s214
00:17:08.720 --> 00:17:17.680
And a big part about putting that into my model is being able to version control it so I can prove that it adds value, not just like vibes and hope that it adds value.

s215
00:17:17.680 --> 00:17:26.319
Um what we're going to do now is like the fun part of this workshop, which is we're going to give you all time to kind of go hands-on with this tool.

s216
00:17:26.319 --> 00:17:29.520
So, there's a little QR code on the screen, you can scan it with a mobile.

s217
00:17:29.520 --> 00:17:40.200
You can also just like enter this [clears throat] little URL into your computer and the uh interface on your laptop will probably be uh more amenable.

s218
00:17:40.200 --> 00:17:44.080
Um and then I'm just going to give you like a little bit of an overview about what we're trying to do here.

s219
00:17:44.080 --> 00:17:56.440
So, the idea is, if you think back to the example that we had before, um you're going to be trying to uh predict like what are company acquisitions right before they happen.

s220
00:17:56.440 --> 00:17:58.520
Um to tell like a little bit of the story here.

s221
00:17:58.520 --> 00:18:03.280
So, let's assume that you're working in some financial institution.

s222
00:18:03.280 --> 00:18:10.960
Uh your boss's boss's boss's boss like went to some AI engineer workshop and they're like, "Hey, I heard this like really, really cool web search feature.

s223
00:18:10.960 --> 00:18:13.720
I think we should present it to our portfolio manager."

s224
00:18:13.720 --> 00:18:15.880
And portfolio manager's like, "Okay, I want to see it.

s225
00:18:15.880 --> 00:18:17.440
You've got 20 minutes.

s226
00:18:17.440 --> 00:18:19.560
Um try and see if you can make it good."

s227
00:18:19.560 --> 00:18:23.120
Um and the most confounding variable here is you don't have like a coding agent.

s228
00:18:23.120 --> 00:18:26.040
So, you have to do it by hand because your usage just ran out or something.

s229
00:18:26.040 --> 00:18:29.520
Um so, that's the kind of context of where we're sitting ourselves at.

s230
00:18:29.520 --> 00:18:37.920
Your task is going to be to find the best two features that like accurately predict a company acquisition.

s231
00:18:37.920 --> 00:18:47.840
Uh and then Access Snapshot is going to use that to go back in time uh 1 day, 7 days, and 30 days before the acquisition and test your feature.

s232
00:18:47.840 --> 00:18:51.880
Um or a random date if they weren't acquired, so there's a few confounding ones in there.

s233
00:18:51.880 --> 00:18:57.200
Um feature is just like a yes/no question um that you believe would predict an acquisition.

s234
00:18:57.200 --> 00:19:04.840
You can test as many as you like, there's a few examples in there, and I'm going to be coming around just to talk with you all um and kind of like help you along the way.

s235
00:19:04.840 --> 00:19:07.360
Um and you can test as many as you like.

s236
00:19:07.360 --> 00:19:18.600
Scoring, you get plus one point if you uh uh if your feature declares that this company would be an acquired uh on 170 30 days before it did.

s237
00:19:18.600 --> 00:19:23.800
You get minus one if you say that it got acquired, uh but it didn't.

s238
00:19:23.800 --> 00:19:26.600
And then, no impact for misses or passes cuz we're very friendly.

s239
00:19:26.600 --> 00:19:29.680
Best score automatically captured, you can try as many as you want.

s240
00:19:29.680 --> 00:19:34.040
I'm going to set us up for let's say like 30 minutes.

s241
00:19:34.040 --> 00:19:38.560
Um so uh we'll kick that off now.

s242
00:19:38.560 --> 00:19:42.440
And you have 30 minutes to try and put in the absolute best feature that you want.

s243
00:19:42.440 --> 00:19:43.160
I'll be coming around.

s244
00:19:43.160 --> 00:19:47.528
Maybe we can put on like a little bit of soft music that kind of gives people the vibe of working in

s245
00:19:47.528 --> 00:19:47.640
[music]

s246
00:19:47.640 --> 00:19:51.960
a portfolio manager company uh and listening to like low-fi beats.

s247
00:19:51.960 --> 00:19:55.200
Uh and I'll come around and talk with you all.

s248
00:19:55.520 --> 00:19:59.480
Okay, I'm just going to do like one like real quick example just to kind of show people how it works.

s249
00:19:59.480 --> 00:20:04.080
If you're already going away and and and winning, keep feel free to keep grinding.

s250
00:20:04.080 --> 00:20:05.680
Um but I'm just going to put one example in here.

s251
00:20:05.680 --> 00:20:07.880
So, you'll see this interface in front of you.

s252
00:20:07.880 --> 00:20:10.960
You have a bunch of um it'll ask you to put in a new feature.

s253
00:20:10.960 --> 00:20:14.640
You can pick one of the examples or type whatever you want and hit run.

s254
00:20:14.640 --> 00:20:18.680
Um and what it'll do is it'll go out, it'll use excess snapshot and search a bunch of them.

s255
00:20:18.680 --> 00:20:25.280
This one was fast because it was cached, so uh that's a performance improvement to show you that we did test this beforehand.

s256
00:20:25.280 --> 00:20:29.760
Um but it'll go down and it'll run that against this set of 10 companies.

s257
00:20:29.760 --> 00:20:33.120
So, within this set of companies, some of them were acquired.

s258
00:20:33.120 --> 00:20:34.320
Some of them were not acquired.

s259
00:20:34.320 --> 00:20:36.240
They're confounding variables.

s260
00:20:36.240 --> 00:20:44.200
Um and the idea is that like we're going to take this particular query, so has this company lost its CEO or CFO or appointed interim leadership.

s261
00:20:44.200 --> 00:20:46.520
And for all of this, we're going to go back and do that version search.

s262
00:20:46.520 --> 00:20:51.360
So, Delivery Hero, we can see, "Hey, um there was an executive leadership turnover.

s263
00:20:51.360 --> 00:20:52.080
Yes."

s264
00:20:52.080 --> 00:20:54.360
And the company did end up being acquired.

s265
00:20:54.360 --> 00:20:56.240
So, hey, maybe this is predictive.

s266
00:20:56.240 --> 00:21:03.920
Except, Quagen, I don't know if I'm pronouncing that correctly, they also had like executive leadership turnover, but they didn't end up being acquired.

s267
00:21:03.920 --> 00:21:05.640
Oh, no, that's a confounding variable.

s268
00:21:05.640 --> 00:21:12.720
So, your task is to try and be like, "Hey, what are the one or two features that I can add to this that will be the most predictive?"

s269
00:21:12.720 --> 00:21:17.040
So, here I'm going to like add one more.

s270
00:21:18.720 --> 00:21:21.760
So, this one is like, "Hey, has it been reported to be exploring a sale?"

s271
00:21:21.760 --> 00:21:24.560
Does adding that improve it or make it worse?

s272
00:21:24.560 --> 00:21:35.880
And in this case, you can see, "Hey, we've picked up a lot of the um uh we've picked up like a lot of the acquisitions, so we're doing pretty well in terms of finding stuff, but also like, "Hey, we're picking up like a lot of false alarms here."

s273
00:21:35.880 --> 00:21:41.000
So, we might need to be more uh constraining in the way that we're combining these two features.

s274
00:21:41.000 --> 00:21:43.840
So, here I've put a very simple like combined thing.

s275
00:21:43.840 --> 00:21:54.120
You can either choose or, which is if either of your two features trigger yes, then we will count that whole thing as being like, "Yeah, you're predicting that it is going to be acquired

s276
00:21:54.120 --> 00:21:55.160
as of that day."

s277
00:21:55.160 --> 00:22:01.320
You can go and, and if I see if I go and here, the score improves a little bit, which means both of these two features need to be true.

s278
00:22:01.320 --> 00:22:06.520
And if both of those two features are true for that company on that day, then you'll trigger yes.

s279
00:22:06.520 --> 00:22:09.000
Um that's just like a PSA.

s280
00:22:09.000 --> 00:22:14.480
I'll go back to the kind of like leadership board there, but feel free to chuck up your hands with any questions as we come through.

s281
00:22:14.480 --> 00:22:16.080
Okay, 10 seconds left.

s282
00:22:16.080 --> 00:22:18.320
If you haven't already hit run, it may be too late.

s283
00:22:18.320 --> 00:22:24.400
Feel free to nobble over the combined and or to try and get your last few results.

s284
00:22:24.400 --> 00:22:24.920
Phew.

s285
00:22:24.920 --> 00:22:26.480
Okay, congratulations.

s286
00:22:26.480 --> 00:22:30.880
So, Zane, John, and Ree, could you come and join me on stage if you won a prize?

s287
00:22:30.880 --> 00:22:33.760
If not, we'll go further down in the list.

s288
00:22:33.760 --> 00:22:36.520
Can we have a big round of applause?

s289
00:22:40.600 --> 00:22:41.360
Come up, don't be shy.

s290
00:22:41.360 --> 00:22:42.480
Don't be shy.

s291
00:22:42.480 --> 00:22:46.600
Brilliant goods and mystery boxes await you.

s292
00:22:47.880 --> 00:22:50.200
Hey, congratulations.

s293
00:22:50.480 --> 00:22:51.320
Congratulations, John.

s294
00:22:51.320 --> 00:22:53.840
Very valuable land.

s295
00:22:56.880 --> 00:23:00.360
Um so, yeah, let me hand these out.

s296
00:23:04.000 --> 00:23:09.280
Thank You're going to line up brilliantly so we can do a photo.

s297
00:23:38.320 --> 00:23:40.600
Okay, we'll take a still out of that video later.

s298
00:23:40.600 --> 00:23:44.360
So, let's just like poke around and be nosy and see like what did our number one person did.

s299
00:23:44.360 --> 00:23:47.920
So, Zane Is this company a publicly listed company?

s300
00:23:47.920 --> 00:23:49.240
Very interesting feature.

s301
00:23:49.240 --> 00:23:55.520
Number two, is this company a company other than Qiagen, Vitris, Bank of Cryptos, and American Electric Power?

s302
00:23:55.520 --> 00:23:57.240
Congratulations.

s303
00:23:57.240 --> 00:24:00.361
You are the reason why I wanted to have a holdout

s304
00:24:00.361 --> 00:24:02.361
[laughter]

s305
00:24:02.560 --> 00:24:09.800
I removed it because I wanted it to be simple, but this is a good example of why you need to do backtesting and why you need to do backtesting with a holdout set.

s306
00:24:09.800 --> 00:24:12.806
Um so, I'm going to take this as a feature, not a bug,

s307
00:24:12.806 --> 00:24:13.000
[laughter]

s308
00:24:13.000 --> 00:24:22.631
of not including a holdout set and why overfitting is very dangerous because I do not think that this particular algorithm would be successful in the future, but who knows?

s309
00:24:22.631 --> 00:24:24.631
[laughter]

s310
00:24:25.680 --> 00:24:26.600
Number two, Ree.

s311
00:24:26.600 --> 00:24:28.526
So, okay, he stole Zane.

s312
00:24:28.526 --> 00:24:29.000
[laughter]

s313
00:24:29.000 --> 00:24:30.920
Maybe it would have been seated nearby.

s314
00:24:30.920 --> 00:24:31.640
And lastly, with John.

s315
00:24:31.640 --> 00:24:43.080
So, within the past 12 months, has an active investor or significant shareholder publicly called for a supported sale of companies as a whole or is company's current share price at least 60% below its high price during the past 5 years or is it named Dominion?

s316
00:24:43.080 --> 00:24:47.760
Also, a little bit of overfitting there, but I would argue it probably performed better.

s317
00:24:47.760 --> 00:24:52.760
Um so, this gives like an example of just like hey, how and why it's important to do backtesting, right?

s318
00:24:52.760 --> 00:24:55.360
Because, you know, this is like a little constrained example.

s319
00:24:55.360 --> 00:25:01.440
I I feel like I'm regretting removing my holdout set, although I do feel like people would have been way more confused if I did do that.

s320
00:25:01.440 --> 00:25:04.920
Um but this gives you an idea of like why it's valuable, right?

s321
00:25:04.920 --> 00:25:12.680
Because if you know what the existing set is, you can you can overfit it, but the idea is like because you have this capability, you can check for that, right?

s322
00:25:12.680 --> 00:25:23.920
You can go back and be like, "Hey, I'm going to go back in my book and be like, hey, I'm going to constrain like these 10 companies that I know something had happened to them, 20 companies that didn't, and train and iterate on my model, right?"

s323
00:25:23.920 --> 00:25:29.200
And make sure that it continues to perform beyond just like the publicly visible sets.

s324
00:25:29.200 --> 00:25:31.400
Um so, let's like recap the highlights.

s325
00:25:31.400 --> 00:25:33.880
I'm also going to Oh my god, there's a lot of people in here.

s326
00:25:33.880 --> 00:25:34.960
I have to scroll very far.

s327
00:25:34.960 --> 00:25:39.120
I'm going to put you all on a uh thank you page.

s328
00:25:39.120 --> 00:25:40.960
I should already be on the thank you page.

s329
00:25:40.960 --> 00:25:43.720
On the thank you page, you've got a link to our docs.

s330
00:25:43.720 --> 00:25:44.800
We love documentation.

s331
00:25:44.800 --> 00:25:46.000
You have a link to our blog.

s332
00:25:46.000 --> 00:25:47.782
It's more readable than the docs

s333
00:25:47.782 --> 00:25:47.920
[snorts]

s334
00:25:47.920 --> 00:25:49.680
and is more like human-facing.

s335
00:25:49.680 --> 00:25:52.760
Uh and lastly, there's a chance to send me an email.

s336
00:25:52.760 --> 00:25:57.480
I really regret that now because there are 100 people in the room, so I'm expecting a lot of emails.

s337
00:25:57.480 --> 00:25:59.320
Um but this is an enterprise feature as well.

s338
00:25:59.320 --> 00:26:07.080
We have some public usage, but if you're interested in having access to this for your team, send me an email and I'll connect you to our enterprise team.

s339
00:26:07.080 --> 00:26:08.960
Um let's just like recap the highlights.

s340
00:26:08.960 --> 00:26:10.640
Number one, we love Exa.

s341
00:26:10.640 --> 00:26:10.960
Why?

s342
00:26:10.960 --> 00:26:13.080
Because it's great agentic web search.

s343
00:26:13.080 --> 00:26:14.280
It's good value.

s344
00:26:14.280 --> 00:26:15.760
It's very cost-effective.

s345
00:26:15.760 --> 00:26:17.400
It's at whatever latency I need.

s346
00:26:17.400 --> 00:26:18.640
I want it in my model.

s347
00:26:18.640 --> 00:26:20.480
Number two, snapshot.

s348
00:26:20.480 --> 00:26:22.400
It lets me backtest my model.

s349
00:26:22.400 --> 00:26:26.960
It lets me see just how good Exa is at improving your model.

s350
00:26:26.960 --> 00:26:33.840
And number three, um consider really adding the holdout set to your competition because otherwise you may overfit.

s351
00:26:33.840 --> 00:26:36.120
Thank you, everyone.